[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786143361779964-25976-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 22:56:01 dist-test-slave-z8x9 krb5kdc[25988](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 22:56:01 dist-test-slave-z8x9 krb5kdc[25988](info): set up 2 sockets
Aug 07 22:56:01 dist-test-slave-z8x9 krb5kdc[25988](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260807 22:56:03.176250 25976 mini_kdc.cc:121] Time spent starting KDC: real 1.294s user 0.001s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786143361779964-25976-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786143361779964-25976-0/krb5kdc/test-user.keytab.
Aug 07 22:56:03 dist-test-slave-z8x9 krb5kdc[25988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143363, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T22:56:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T22:56:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.94.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.62.keytab.
Entry for principal kudu/127.25.94.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.62.keytab.
I20260807 22:56:03.394205 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786143361779964-25976-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786143361779964-25976-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786143361779964-25976-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.62:36767
--webserver_interface=127.25.94.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.62.keytab
--principal=kudu/127.25.94.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.20:38275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.94.62:36767
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786143361779964-25976-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786143361779964-25976-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786143361779964-25976-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:56:03.768159 26009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:56:03.768538 26009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:56:03.768707 26009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:56:03.775997 26009 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 22:56:03.776108 26009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:56:03.776211 26009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:56:03.776309 26009 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 22:56:03.776398 26009 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 22:56:03.786857 26009 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:38275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786143361779964-25976-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786143361779964-25976-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.94.62:36767
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.62.keytab
--principal=kudu/127.25.94.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.62:36767
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.94.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786143361779964-25976-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 22:56:03.788790 26009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:56:03.790585 26009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:56:03.811396 26021 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:56:03.811432 26024 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:56:03.813755 26020 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:56:03.815462 26009 server_base.cc:1034] running on GCE node
I20260807 22:56:03.816460 26009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:56:03.818789 26009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:56:03.820067 26009 hybrid_clock.cc:648] HybridClock initialized: now 1786143363819980 us; error 89 us; skew 500 ppm
I20260807 22:56:03.820607 26009 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 22:56:03 dist-test-slave-z8x9 krb5kdc[25988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143363, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:56:03.824553 26009 init.cc:377] Logged in from keytab as kudu/127.25.94.62@KRBTEST.COM (short username kudu)
I20260807 22:56:03.827001 26009 webserver.cc:460] Webserver started at http://127.25.94.62:41469/ using document root <none> and password file <none>
I20260807 22:56:03.828116 26009 fs_manager.cc:359] Metadata directory not provided
I20260807 22:56:03.828280 26009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:56:03.828768 26009 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:56:03.832818 26009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786143361779964-25976-0/minicluster-data/master-0/data/instance:
uuid: "2ed25a395e3c4307821a9e609f1a5317"
format_stamp: "Formatted at 2026-08-07 22:56:03 on dist-test-slave-z8x9"
server_key: "156f6260b948b770e52b6e73b5c0b692"
server_key_iv: "1959ab9fcbbc9e8f06bafded7b17b22c"
server_key_version: "encryptionkey@0"
I20260807 22:56:03.833671 26009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786143361779964-25976-0/minicluster-data/master-0/wal/instance:
uuid: "2ed25a395e3c4307821a9e609f1a5317"
format_stamp: "Formatted at 2026-08-07 22:56:03 on dist-test-slave-z8x9"
server_key: "156f6260b948b770e52b6e73b5c0b692"
server_key_iv: "1959ab9fcbbc9e8f06bafded7b17b22c"
server_key_version: "encryptionkey@0"
I20260807 22:56:03.841827 26009 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.007s sys 0.000s
I20260807 22:56:03.847280 26032 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:56:03.849152 26009 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260807 22:56:03.849516 26009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786143361779964-25976-0/minicluster-data/master-0/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786143361779964-25976-0/minicluster-data/master-0/wal
uuid: "2ed25a395e3c4307821a9e609f1a5317"
format_stamp: "Formatted at 2026-08-07 22:56:03 on dist-test-slave-z8x9"
server_key: "156f6260b948b770e52b6e73b5c0b692"
server_key_iv: "1959ab9fcbbc9e8f06bafded7b17b22c"
server_key_version: "encryptionkey@0"
I20260807 22:56:03.849781 26009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786143361779964-25976-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786143361779964-25976-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786143361779964-25976-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:56:03.875612 26009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:56:03.881635 26009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:56:03.882056 26009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:56:03.905201 26009 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.62:36767
I20260807 22:56:03.905304 26113 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.62:36767 every 8 connection(s)
I20260807 22:56:03.907557 26009 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
I20260807 22:56:03.912310 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 26009
I20260807 22:56:03.912950 25976 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786143361779964-25976-0/minicluster-data/master-0/wal/instance
I20260807 22:56:03.916117 26114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:56:03.916563 25976 external_mini_cluster.cc:1428] Setting key 3f45484a93629d5acf0144599fea9cb8
I20260807 22:56:03.938299 26114 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ed25a395e3c4307821a9e609f1a5317: Bootstrap starting.
I20260807 22:56:03.944907 26114 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2ed25a395e3c4307821a9e609f1a5317: Neither blocks nor log segments found. Creating new log.
I20260807 22:56:03.947127 26114 log.cc:824] T 00000000000000000000000000000000 P 2ed25a395e3c4307821a9e609f1a5317: Log is configured to *not* fsync() on all Append() calls
I20260807 22:56:03.952132 26114 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ed25a395e3c4307821a9e609f1a5317: No bootstrap required, opened a new log
Aug 07 22:56:03 dist-test-slave-z8x9 krb5kdc[25988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143363, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.62@KRBTEST.COM
I20260807 22:56:03.965106 26114 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2ed25a395e3c4307821a9e609f1a5317 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ed25a395e3c4307821a9e609f1a5317" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 36767 } }
I20260807 22:56:03.965819 26114 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2ed25a395e3c4307821a9e609f1a5317 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:56:03.966090 26114 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2ed25a395e3c4307821a9e609f1a5317 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ed25a395e3c4307821a9e609f1a5317, State: Initialized, Role: FOLLOWER
I20260807 22:56:03.967705 26114 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2ed25a395e3c4307821a9e609f1a5317 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ed25a395e3c4307821a9e609f1a5317" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 36767 } }
I20260807 22:56:03.968187 26114 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2ed25a395e3c4307821a9e609f1a5317 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 22:56:03.968544 26114 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2ed25a395e3c4307821a9e609f1a5317 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 22:56:03.968811 26114 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2ed25a395e3c4307821a9e609f1a5317 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:56:03.972266 26114 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2ed25a395e3c4307821a9e609f1a5317 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ed25a395e3c4307821a9e609f1a5317" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 36767 } }
I20260807 22:56:03.973412 26114 leader_election.cc:302] T 00000000000000000000000000000000 P 2ed25a395e3c4307821a9e609f1a5317 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2ed25a395e3c4307821a9e609f1a5317; no voters:
I20260807 22:56:03.974474 26114 leader_election.cc:288] T 00000000000000000000000000000000 P 2ed25a395e3c4307821a9e609f1a5317 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 22:56:03.975337 26121 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2ed25a395e3c4307821a9e609f1a5317 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:56:03.976280 26121 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2ed25a395e3c4307821a9e609f1a5317 [term 1 LEADER]: Becoming Leader. State: Replica: 2ed25a395e3c4307821a9e609f1a5317, State: Running, Role: LEADER
I20260807 22:56:03.976686 26116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:03.924887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51688 (local address 127.25.94.62:36767)
0807 22:56:03.928606 (+ 3719us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:03.928619 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:03.928665 (+ 46us) server_negotiation.cc:407] Connection header received
0807 22:56:03.942739 (+ 14074us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:03.942762 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:03.943368 (+ 606us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:03.943748 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:56:03.948877 (+ 5129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:03.950374 (+ 1497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:03.953692 (+ 3318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:03.954530 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:03.954760 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:03.963415 (+ 8655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:56:03.963498 (+ 83us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:56:03.963701 (+ 203us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:56:03.963785 (+ 84us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:56:03.968785 (+ 5000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:03.970080 (+ 1295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:03.970093 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:03.970106 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:03.970390 (+ 284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:03.970988 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:03.970995 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:03.970999 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:03.972021 (+ 1022us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:56:03.972509 (+ 488us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:03.973804 (+ 1295us) server_negotiation.cc:299] Negotiation successful
0807 22:56:03.974336 (+ 532us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3438,"thread_start_us":3037,"threads_started":1}
I20260807 22:56:03.977276 26121 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2ed25a395e3c4307821a9e609f1a5317 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ed25a395e3c4307821a9e609f1a5317" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 36767 } }
I20260807 22:56:03.978300 26114 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2ed25a395e3c4307821a9e609f1a5317 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 22:56:03.984000 26123 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ed25a395e3c4307821a9e609f1a5317 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2ed25a395e3c4307821a9e609f1a5317" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ed25a395e3c4307821a9e609f1a5317" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 36767 } } }
I20260807 22:56:03.984354 26123 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ed25a395e3c4307821a9e609f1a5317 [sys.catalog]: This master's current role is: LEADER
I20260807 22:56:03.985611 26134 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 22:56:03.985509 26125 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ed25a395e3c4307821a9e609f1a5317 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2ed25a395e3c4307821a9e609f1a5317. Latest consensus state: current_term: 1 leader_uuid: "2ed25a395e3c4307821a9e609f1a5317" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ed25a395e3c4307821a9e609f1a5317" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 36767 } } }
I20260807 22:56:03.985970 26125 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ed25a395e3c4307821a9e609f1a5317 [sys.catalog]: This master's current role is: LEADER
I20260807 22:56:03.994984 26134 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 22:56:04.016893 26134 catalog_manager.cc:1347] Generated new cluster ID: 991168925db04990b1f35798b0d396d8
I20260807 22:56:04.017040 26134 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 22:56:04.035696 26134 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 22:56:04.037959 26134 catalog_manager.cc:1504] Loading token signing keys...
I20260807 22:56:04.050298 26134 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2ed25a395e3c4307821a9e609f1a5317: Generated new TSK 0
I20260807 22:56:04.051623 26134 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.94.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.1.keytab.
Entry for principal kudu/127.25.94.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.1.keytab.
I20260807 22:56:04.109500 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786143361779964-25976-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786143361779964-25976-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786143361779964-25976-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786143361779964-25976-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.1:0
--local_ip_for_outbound_sockets=127.25.94.1
--webserver_interface=127.25.94.1
--webserver_port=0
--tserver_master_addrs=127.25.94.62:36767
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.1.keytab
--principal=kudu/127.25.94.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.20:38275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786143361779964-25976-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786143361779964-25976-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786143361779964-25976-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:56:04.472397 26155 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:56:04.472765 26155 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:56:04.472952 26155 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:56:04.481302 26155 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:56:04.481467 26155 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:56:04.481647 26155 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.1
I20260807 22:56:04.492142 26155 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:38275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786143361779964-25976-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786143361779964-25976-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.1.keytab
--principal=kudu/127.25.94.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786143361779964-25976-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.94.1
--webserver_port=0
--tserver_master_addrs=127.25.94.62:36767
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.1
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786143361779964-25976-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 22:56:04.493614 26155 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:56:04.495059 26155 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:56:04.509642 26166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:56:04.509675 26167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:56:04.509599 26170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:56:04.510679 26155 server_base.cc:1034] running on GCE node
I20260807 22:56:04.511492 26155 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:56:04.512421 26155 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:56:04.513623 26155 hybrid_clock.cc:648] HybridClock initialized: now 1786143364513559 us; error 57 us; skew 500 ppm
I20260807 22:56:04.513957 26155 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 22:56:04 dist-test-slave-z8x9 krb5kdc[25988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143364, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:56:04.517141 26155 init.cc:377] Logged in from keytab as kudu/127.25.94.1@KRBTEST.COM (short username kudu)
I20260807 22:56:04.518776 26155 webserver.cc:460] Webserver started at http://127.25.94.1:33757/ using document root <none> and password file <none>
I20260807 22:56:04.519497 26155 fs_manager.cc:359] Metadata directory not provided
I20260807 22:56:04.519649 26155 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:56:04.519995 26155 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:56:04.522991 26155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786143361779964-25976-0/minicluster-data/ts-0/data/instance:
uuid: "825b4f689a20420f8f982ebc065809ed"
format_stamp: "Formatted at 2026-08-07 22:56:04 on dist-test-slave-z8x9"
server_key: "c5edca73b284ff5ea6ecd8cf2667b925"
server_key_iv: "e5e452fb391dc833005f305dfeb71c21"
server_key_version: "encryptionkey@0"
I20260807 22:56:04.523772 26155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786143361779964-25976-0/minicluster-data/ts-0/wal/instance:
uuid: "825b4f689a20420f8f982ebc065809ed"
format_stamp: "Formatted at 2026-08-07 22:56:04 on dist-test-slave-z8x9"
server_key: "c5edca73b284ff5ea6ecd8cf2667b925"
server_key_iv: "e5e452fb391dc833005f305dfeb71c21"
server_key_version: "encryptionkey@0"
I20260807 22:56:04.529414 26155 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260807 22:56:04.533052 26187 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:56:04.534493 26155 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 22:56:04.534684 26155 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786143361779964-25976-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786143361779964-25976-0/minicluster-data/ts-0/wal
uuid: "825b4f689a20420f8f982ebc065809ed"
format_stamp: "Formatted at 2026-08-07 22:56:04 on dist-test-slave-z8x9"
server_key: "c5edca73b284ff5ea6ecd8cf2667b925"
server_key_iv: "e5e452fb391dc833005f305dfeb71c21"
server_key_version: "encryptionkey@0"
I20260807 22:56:04.534898 26155 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786143361779964-25976-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786143361779964-25976-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786143361779964-25976-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:56:04.561805 26155 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:56:04.566099 26155 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:56:04.566416 26155 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:56:04.567480 26155 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:56:04.569140 26155 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:56:04.569272 26155 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:56:04.569494 26155 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:56:04.569576 26155 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:56:04.607292 26155 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.1:32999
I20260807 22:56:04.607347 26375 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.1:32999 every 8 connection(s)
I20260807 22:56:04.608692 26155 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786143361779964-25976-0/minicluster-data/ts-0/data/info.pb
I20260807 22:56:04.610716 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 26155
I20260807 22:56:04.610968 25976 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786143361779964-25976-0/minicluster-data/ts-0/wal/instance
I20260807 22:56:04.611589 25976 external_mini_cluster.cc:1428] Setting key efc7e05998aed5748cc6f2e50c4d930f
Aug 07 22:56:04 dist-test-slave-z8x9 krb5kdc[25988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143364, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.1@KRBTEST.COM for kudu/127.25.94.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.25.94.2@KRBTEST.COM; defaulting to no policy
I20260807 22:56:04.633965 26380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:04.612249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.1:56141 (local address 127.25.94.62:36767)
0807 22:56:04.612801 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:04.612806 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:04.614052 (+ 1246us) server_negotiation.cc:407] Connection header received
0807 22:56:04.615971 (+ 1919us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:04.615980 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:04.616079 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:04.616207 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:56:04.617903 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:04.619198 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:04.621728 (+ 2530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:04.622559 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:04.622719 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:04.626746 (+ 4027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:56:04.626754 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:56:04.626760 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:56:04.626810 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:56:04.630655 (+ 3845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:04.631504 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:04.631512 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:04.631516 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:04.631680 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:04.632176 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:04.632184 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:04.632190 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:04.632502 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:56:04.632618 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:04.633528 (+ 910us) server_negotiation.cc:299] Negotiation successful
0807 22:56:04.633703 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":256,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.2@KRBTEST.COM" created.
I20260807 22:56:04.635193 26383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:04.612882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:36767 (local address 127.25.94.1:56141)
0807 22:56:04.613872 (+ 990us) negotiation.cc:107] Waiting for socket to connect
0807 22:56:04.613935 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0807 22:56:04.615660 (+ 1725us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:56:04.616450 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:56:04.616458 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:56:04.617019 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:56:04.617728 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:04.617747 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:04.619415 (+ 1668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:04.619424 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:04.621557 (+ 2133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:04.621577 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:04.622815 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:04.622820 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:04.623018 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:04.623799 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:56:04.623823 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:56:04.626483 (+ 2660us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:56:04.630829 (+ 4346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:56:04.630837 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:56:04.630850 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:56:04.631353 (+ 503us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:56:04.631802 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:56:04.631805 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:56:04.631808 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:56:04.632042 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:56:04.632624 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:56:04.632630 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:56:04.633011 (+ 381us) client_negotiation.cc:769] Sending connection context
0807 22:56:04.633403 (+ 392us) client_negotiation.cc:240] Negotiation successful
0807 22:56:04.633815 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":691,"thread_start_us":456,"threads_started":1}
I20260807 22:56:04.637037 26378 heartbeater.cc:347] Connected to a master server at 127.25.94.62:36767
I20260807 22:56:04.637566 26378 heartbeater.cc:464] Registering TS with master...
I20260807 22:56:04.638552 26378 heartbeater.cc:511] Master 127.25.94.62:36767 requested a full tablet report, sending...
I20260807 22:56:04.642424 26056 ts_manager.cc:194] Registered new tserver with Master: 825b4f689a20420f8f982ebc065809ed (127.25.94.1:32999)
I20260807 22:56:04.644485 26056 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.94.1@KRBTEST.COM'} at 127.25.94.1:56141
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.2.keytab.
Entry for principal kudu/127.25.94.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.2.keytab.
I20260807 22:56:04.658658 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786143361779964-25976-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786143361779964-25976-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786143361779964-25976-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786143361779964-25976-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.2:0
--local_ip_for_outbound_sockets=127.25.94.2
--webserver_interface=127.25.94.2
--webserver_port=0
--tserver_master_addrs=127.25.94.62:36767
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.2.keytab
--principal=kudu/127.25.94.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.20:38275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786143361779964-25976-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786143361779964-25976-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786143361779964-25976-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:56:05.010614 26387 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:56:05.010937 26387 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:56:05.011039 26387 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:56:05.018620 26387 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:56:05.018751 26387 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:56:05.018868 26387 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.2
I20260807 22:56:05.028808 26387 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:38275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786143361779964-25976-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786143361779964-25976-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.2.keytab
--principal=kudu/127.25.94.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786143361779964-25976-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.94.2
--webserver_port=0
--tserver_master_addrs=127.25.94.62:36767
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.2
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786143361779964-25976-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 22:56:05.030189 26387 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:56:05.031555 26387 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:56:05.045132 26402 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:56:05.045747 26398 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:56:05.045934 26400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:56:05.046404 26387 server_base.cc:1034] running on GCE node
I20260807 22:56:05.047049 26387 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:56:05.048049 26387 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:56:05.049284 26387 hybrid_clock.cc:648] HybridClock initialized: now 1786143365049218 us; error 59 us; skew 500 ppm
I20260807 22:56:05.049597 26387 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 22:56:05 dist-test-slave-z8x9 krb5kdc[25988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143365, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:56:05.052856 26387 init.cc:377] Logged in from keytab as kudu/127.25.94.2@KRBTEST.COM (short username kudu)
I20260807 22:56:05.054385 26387 webserver.cc:460] Webserver started at http://127.25.94.2:45873/ using document root <none> and password file <none>
I20260807 22:56:05.055068 26387 fs_manager.cc:359] Metadata directory not provided
I20260807 22:56:05.055177 26387 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:56:05.055491 26387 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:56:05.058578 26387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786143361779964-25976-0/minicluster-data/ts-1/data/instance:
uuid: "c04ce3464f524d228f3f9fc6b4a90cf3"
format_stamp: "Formatted at 2026-08-07 22:56:05 on dist-test-slave-z8x9"
server_key: "f54e5f5fc4f55604b0e6aea4115a53ef"
server_key_iv: "a6033627ddf6e404055974bde13ea83c"
server_key_version: "encryptionkey@0"
I20260807 22:56:05.059430 26387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786143361779964-25976-0/minicluster-data/ts-1/wal/instance:
uuid: "c04ce3464f524d228f3f9fc6b4a90cf3"
format_stamp: "Formatted at 2026-08-07 22:56:05 on dist-test-slave-z8x9"
server_key: "f54e5f5fc4f55604b0e6aea4115a53ef"
server_key_iv: "a6033627ddf6e404055974bde13ea83c"
server_key_version: "encryptionkey@0"
I20260807 22:56:05.064560 26387 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260807 22:56:05.068121 26414 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:56:05.069581 26387 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 22:56:05.069763 26387 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786143361779964-25976-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786143361779964-25976-0/minicluster-data/ts-1/wal
uuid: "c04ce3464f524d228f3f9fc6b4a90cf3"
format_stamp: "Formatted at 2026-08-07 22:56:05 on dist-test-slave-z8x9"
server_key: "f54e5f5fc4f55604b0e6aea4115a53ef"
server_key_iv: "a6033627ddf6e404055974bde13ea83c"
server_key_version: "encryptionkey@0"
I20260807 22:56:05.069964 26387 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786143361779964-25976-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786143361779964-25976-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786143361779964-25976-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:56:05.092128 26387 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:56:05.097260 26387 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:56:05.097584 26387 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:56:05.098652 26387 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:56:05.100114 26387 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:56:05.100232 26387 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:56:05.100382 26387 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:56:05.100458 26387 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:56:05.139351 26387 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.2:33391
I20260807 22:56:05.139623 26585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.2:33391 every 8 connection(s)
I20260807 22:56:05.140892 26387 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786143361779964-25976-0/minicluster-data/ts-1/data/info.pb
I20260807 22:56:05.151403 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 26387
I20260807 22:56:05.151654 25976 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786143361779964-25976-0/minicluster-data/ts-1/wal/instance
I20260807 22:56:05.152228 25976 external_mini_cluster.cc:1428] Setting key df647575eedf7c2e9acc848e3b7079c5
Aug 07 22:56:05 dist-test-slave-z8x9 krb5kdc[25988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143365, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.2@KRBTEST.COM for kudu/127.25.94.62@KRBTEST.COM
I20260807 22:56:05.167789 26590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:05.143891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.2:41459 (local address 127.25.94.62:36767)
0807 22:56:05.144394 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:05.144401 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:05.145544 (+ 1143us) server_negotiation.cc:407] Connection header received
0807 22:56:05.147711 (+ 2167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:05.147717 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:05.147801 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:05.147913 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:56:05.149905 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:05.150997 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:05.153581 (+ 2584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:05.154344 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:05.154588 (+ 244us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:05.159369 (+ 4781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:56:05.159380 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:56:05.159388 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:56:05.159443 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:56:05.162810 (+ 3367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:05.164012 (+ 1202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:05.164017 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:05.164020 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:05.164179 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:05.164748 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:05.164753 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:05.164756 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:05.165221 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:56:05.165394 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:05.166974 (+ 1580us) server_negotiation.cc:299] Negotiation successful
0807 22:56:05.167292 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":234,"threads_started":1}
I20260807 22:56:05.169466 26591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:05.144441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:36767 (local address 127.25.94.2:41459)
0807 22:56:05.145374 (+ 933us) negotiation.cc:107] Waiting for socket to connect
0807 22:56:05.145420 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0807 22:56:05.147346 (+ 1926us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:56:05.148237 (+ 891us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:56:05.148248 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:56:05.148905 (+ 657us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:56:05.149668 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:05.149689 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:05.151220 (+ 1531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:05.151227 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:05.153360 (+ 2133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:05.153379 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:05.154775 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:05.154782 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:05.155135 (+ 353us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:05.156167 (+ 1032us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:56:05.156200 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:56:05.159023 (+ 2823us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:56:05.163023 (+ 4000us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:56:05.163037 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:56:05.163061 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:56:05.163843 (+ 782us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:56:05.164299 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:56:05.164307 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:56:05.164314 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:56:05.164600 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:56:05.165424 (+ 824us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:56:05.165476 (+ 52us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:56:05.166408 (+ 932us) client_negotiation.cc:769] Sending connection context
0807 22:56:05.166808 (+ 400us) client_negotiation.cc:240] Negotiation successful
0807 22:56:05.167487 (+ 679us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":332,"threads_started":1}
I20260807 22:56:05.171530 26587 heartbeater.cc:347] Connected to a master server at 127.25.94.62:36767
I20260807 22:56:05.171926 26587 heartbeater.cc:464] Registering TS with master...
I20260807 22:56:05.172976 26587 heartbeater.cc:511] Master 127.25.94.62:36767 requested a full tablet report, sending...
I20260807 22:56:05.177316 26056 ts_manager.cc:194] Registered new tserver with Master: c04ce3464f524d228f3f9fc6b4a90cf3 (127.25.94.2:33391)
WARNING: no policy specified for kudu/127.25.94.3@KRBTEST.COM; defaulting to no policy
I20260807 22:56:05.178831 26056 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.94.2@KRBTEST.COM'} at 127.25.94.2:41459
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.3.keytab.
Entry for principal kudu/127.25.94.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.3.keytab.
I20260807 22:56:05.209643 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786143361779964-25976-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786143361779964-25976-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786143361779964-25976-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786143361779964-25976-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.3:0
--local_ip_for_outbound_sockets=127.25.94.3
--webserver_interface=127.25.94.3
--webserver_port=0
--tserver_master_addrs=127.25.94.62:36767
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.3.keytab
--principal=kudu/127.25.94.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.20:38275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786143361779964-25976-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786143361779964-25976-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786143361779964-25976-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:56:05.565341 26596 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:56:05.565653 26596 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:56:05.565814 26596 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:56:05.573416 26596 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:56:05.573549 26596 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:56:05.573710 26596 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.3
I20260807 22:56:05.584481 26596 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:38275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786143361779964-25976-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786143361779964-25976-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.3.keytab
--principal=kudu/127.25.94.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786143361779964-25976-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.94.3
--webserver_port=0
--tserver_master_addrs=127.25.94.62:36767
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.3
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786143361779964-25976-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 22:56:05.585901 26596 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:56:05.587296 26596 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:56:05.601087 26606 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:56:05.601138 26607 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:56:05.601063 26611 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:56:05.602257 26596 server_base.cc:1034] running on GCE node
I20260807 22:56:05.602926 26596 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:56:05.603916 26596 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:56:05.605216 26596 hybrid_clock.cc:648] HybridClock initialized: now 1786143365605109 us; error 54 us; skew 500 ppm
I20260807 22:56:05.605576 26596 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 22:56:05 dist-test-slave-z8x9 krb5kdc[25988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143365, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:56:05.609724 26596 init.cc:377] Logged in from keytab as kudu/127.25.94.3@KRBTEST.COM (short username kudu)
I20260807 22:56:05.613847 26596 webserver.cc:460] Webserver started at http://127.25.94.3:33435/ using document root <none> and password file <none>
I20260807 22:56:05.614557 26596 fs_manager.cc:359] Metadata directory not provided
I20260807 22:56:05.614704 26596 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:56:05.615036 26596 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:56:05.618135 26596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786143361779964-25976-0/minicluster-data/ts-2/data/instance:
uuid: "6ddc5bae2b634df097debe8aa0744e9a"
format_stamp: "Formatted at 2026-08-07 22:56:05 on dist-test-slave-z8x9"
server_key: "f43486c9581eb9f51eaf14d743f75067"
server_key_iv: "869dce46e77c2c35dd1203e83a96f092"
server_key_version: "encryptionkey@0"
I20260807 22:56:05.618968 26596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786143361779964-25976-0/minicluster-data/ts-2/wal/instance:
uuid: "6ddc5bae2b634df097debe8aa0744e9a"
format_stamp: "Formatted at 2026-08-07 22:56:05 on dist-test-slave-z8x9"
server_key: "f43486c9581eb9f51eaf14d743f75067"
server_key_iv: "869dce46e77c2c35dd1203e83a96f092"
server_key_version: "encryptionkey@0"
I20260807 22:56:05.624601 26596 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260807 22:56:05.628531 26625 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:56:05.630343 26596 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 22:56:05.630560 26596 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786143361779964-25976-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786143361779964-25976-0/minicluster-data/ts-2/wal
uuid: "6ddc5bae2b634df097debe8aa0744e9a"
format_stamp: "Formatted at 2026-08-07 22:56:05 on dist-test-slave-z8x9"
server_key: "f43486c9581eb9f51eaf14d743f75067"
server_key_iv: "869dce46e77c2c35dd1203e83a96f092"
server_key_version: "encryptionkey@0"
I20260807 22:56:05.630779 26596 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786143361779964-25976-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786143361779964-25976-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786143361779964-25976-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:56:05.648165 26378 heartbeater.cc:503] Master 127.25.94.62:36767 was elected leader, sending a full tablet report...
I20260807 22:56:05.657248 26596 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:56:05.662071 26596 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:56:05.662396 26596 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:56:05.663471 26596 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:56:05.664896 26596 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:56:05.664999 26596 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:56:05.665158 26596 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:56:05.665254 26596 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260807 22:56:05.711131 26596 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.3:38365
I20260807 22:56:05.711223 26800 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.3:38365 every 8 connection(s)
I20260807 22:56:05.712487 26596 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786143361779964-25976-0/minicluster-data/ts-2/data/info.pb
I20260807 22:56:05.721134 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 26596
I20260807 22:56:05.721431 25976 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786143361779964-25976-0/minicluster-data/ts-2/wal/instance
I20260807 22:56:05.722014 25976 external_mini_cluster.cc:1428] Setting key de1eace3723493df34853efd69dd7a4d
Aug 07 22:56:05 dist-test-slave-z8x9 krb5kdc[25988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143365, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.3@KRBTEST.COM for kudu/127.25.94.62@KRBTEST.COM
I20260807 22:56:05.736001 26807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:05.715192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.3:39015 (local address 127.25.94.62:36767)
0807 22:56:05.715689 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:05.715696 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:05.716646 (+ 950us) server_negotiation.cc:407] Connection header received
0807 22:56:05.718565 (+ 1919us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:05.718570 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:05.718657 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:05.718785 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:56:05.720315 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:05.721334 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:05.723556 (+ 2222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:05.724310 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:05.724461 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:05.728727 (+ 4266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:56:05.728732 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:56:05.728736 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:56:05.728769 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:56:05.732316 (+ 3547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:05.733060 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:05.733067 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:05.733070 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:05.733295 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:05.733740 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:05.733745 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:05.733749 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:05.734084 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:56:05.734283 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:05.735076 (+ 793us) server_negotiation.cc:299] Negotiation successful
0807 22:56:05.735357 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":224,"threads_started":1}
I20260807 22:56:05.736905 26809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:05.715745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:36767 (local address 127.25.94.3:39015)
0807 22:56:05.716486 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0807 22:56:05.716531 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0807 22:56:05.718274 (+ 1743us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:56:05.718995 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:56:05.719003 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:56:05.719575 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:56:05.720153 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:05.720171 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:05.721501 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:05.721509 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:05.723410 (+ 1901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:05.723422 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:05.724558 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:05.724582 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:05.724925 (+ 343us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:05.726011 (+ 1086us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:56:05.726035 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:56:05.728510 (+ 2475us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:56:05.732460 (+ 3950us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:56:05.732469 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:56:05.732487 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:56:05.732933 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:56:05.733415 (+ 482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:56:05.733421 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:56:05.733424 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:56:05.733617 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:56:05.734236 (+ 619us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:56:05.734245 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:56:05.734611 (+ 366us) client_negotiation.cc:769] Sending connection context
0807 22:56:05.734946 (+ 335us) client_negotiation.cc:240] Negotiation successful
0807 22:56:05.735537 (+ 591us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":288,"threads_started":1}
I20260807 22:56:05.738636 26801 heartbeater.cc:347] Connected to a master server at 127.25.94.62:36767
I20260807 22:56:05.739099 26801 heartbeater.cc:464] Registering TS with master...
I20260807 22:56:05.740243 26801 heartbeater.cc:511] Master 127.25.94.62:36767 requested a full tablet report, sending...
I20260807 22:56:05.742059 26055 ts_manager.cc:194] Registered new tserver with Master: 6ddc5bae2b634df097debe8aa0744e9a (127.25.94.3:38365)
I20260807 22:56:05.742978 26055 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.94.3@KRBTEST.COM'} at 127.25.94.3:39015
I20260807 22:56:05.752285 25976 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 07 22:56:05 dist-test-slave-z8x9 krb5kdc[25988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143365, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 07 22:56:05 dist-test-slave-z8x9 krb5kdc[25988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143365, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.25.94.1@KRBTEST.COM
I20260807 22:56:05.801216 26817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:05.785295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33630 (local address 127.25.94.1:32999)
0807 22:56:05.786046 (+ 751us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:05.786054 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:05.786081 (+ 27us) server_negotiation.cc:407] Connection header received
0807 22:56:05.786145 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:05.786151 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:05.786370 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:05.786502 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:56:05.787937 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:05.788943 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:05.790332 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:05.790915 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:05.791015 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:05.795072 (+ 4057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:56:05.795087 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:56:05.795105 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:56:05.795162 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:56:05.798433 (+ 3271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:05.799216 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:05.799223 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:05.799225 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:05.799391 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:05.799766 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:05.799769 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:05.799771 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:05.800134 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:56:05.800463 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:05.800660 (+ 197us) server_negotiation.cc:299] Negotiation successful
0807 22:56:05.800871 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":561,"thread_start_us":366,"threads_started":1}
W20260807 22:56:05.802618 26233 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33630
Aug 07 22:56:05 dist-test-slave-z8x9 krb5kdc[25988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143365, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 07 22:56:05 dist-test-slave-z8x9 krb5kdc[25988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143365, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.1@KRBTEST.COM
I20260807 22:56:05.855926 26817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:05.836386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33636 (local address 127.25.94.1:32999)
0807 22:56:05.836641 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:05.836651 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:05.837002 (+ 351us) server_negotiation.cc:407] Connection header received
0807 22:56:05.837278 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:05.837283 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:05.837386 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:05.837497 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:56:05.838899 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:05.839594 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:05.840849 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:05.841578 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:05.841731 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:05.847554 (+ 5823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:56:05.847563 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:56:05.847570 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:56:05.847621 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:56:05.851835 (+ 4214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:05.852885 (+ 1050us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:05.852905 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:05.852911 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:05.853089 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:05.853919 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:05.853926 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:05.853931 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:05.854303 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:56:05.854583 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:05.855409 (+ 826us) server_negotiation.cc:299] Negotiation successful
0807 22:56:05.855686 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260807 22:56:05.858480 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 26155
I20260807 22:56:05.879566 26817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:05.860006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33642 (local address 127.25.94.1:32999)
0807 22:56:05.860216 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:05.860222 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:05.860449 (+ 227us) server_negotiation.cc:407] Connection header received
0807 22:56:05.860646 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:05.860653 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:05.860746 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:05.860841 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:56:05.862654 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:05.863755 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:05.866429 (+ 2674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:05.866986 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:05.867131 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:05.871120 (+ 3989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:56:05.871125 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:56:05.871128 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:56:05.871161 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:56:05.875222 (+ 4061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:05.876484 (+ 1262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:05.876494 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:05.876500 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:05.876830 (+ 330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:05.877564 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:05.877572 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:05.877579 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:05.878032 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:56:05.878370 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:05.878714 (+ 344us) server_negotiation.cc:299] Negotiation successful
0807 22:56:05.878947 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260807 22:56:05.880827 26369 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 22:56:05.981415 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 26155
I20260807 22:56:06.004078 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 26387
Aug 07 22:56:06 dist-test-slave-z8x9 krb5kdc[25988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143365, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.2@KRBTEST.COM
I20260807 22:56:06.020915 26831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:06.005008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33914 (local address 127.25.94.2:33391)
0807 22:56:06.005678 (+ 670us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:06.005686 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:06.005713 (+ 27us) server_negotiation.cc:407] Connection header received
0807 22:56:06.005775 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:06.005782 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:06.005984 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:06.006125 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:56:06.007317 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:06.008440 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:06.009968 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:06.010583 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:06.010698 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:06.015182 (+ 4484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:56:06.015224 (+ 42us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:56:06.015241 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:56:06.015286 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:56:06.018365 (+ 3079us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:06.018998 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:06.019005 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:06.019007 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:06.019134 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:06.019546 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:06.019551 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:06.019553 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:06.019922 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:56:06.020224 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:06.020381 (+ 157us) server_negotiation.cc:299] Negotiation successful
0807 22:56:06.020586 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":510,"thread_start_us":398,"threads_started":1}
I20260807 22:56:06.022333 26580 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 22:56:06.109154 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 26387
I20260807 22:56:06.137378 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 26596
Aug 07 22:56:06 dist-test-slave-z8x9 krb5kdc[25988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143365, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.3@KRBTEST.COM
I20260807 22:56:06.156919 26837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:06.138580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54988 (local address 127.25.94.3:38365)
0807 22:56:06.139257 (+ 677us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:06.139265 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:06.139299 (+ 34us) server_negotiation.cc:407] Connection header received
0807 22:56:06.139366 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:06.139373 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:06.139606 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:06.139792 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:56:06.141155 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:06.142317 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:06.143673 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:06.144262 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:06.144413 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:06.149528 (+ 5115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:56:06.149545 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:56:06.149562 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:56:06.149609 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:56:06.153379 (+ 3770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:06.154411 (+ 1032us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:06.154428 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:06.154430 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:06.154614 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:06.155071 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:06.155075 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:06.155076 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:06.155503 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:56:06.155846 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:06.156200 (+ 354us) server_negotiation.cc:299] Negotiation successful
0807 22:56:06.156526 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":488,"thread_start_us":304,"threads_started":1}
I20260807 22:56:06.158613 26794 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 22:56:06.244123 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 26596
I20260807 22:56:06.266759 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 26009
I20260807 22:56:06.268157 26108 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 22:56:06.373143 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 26009
2026-08-07T22:56:06Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (4556 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786143361779964-25976-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 22:56:06 dist-test-slave-z8x9 krb5kdc[26849](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 22:56:06 dist-test-slave-z8x9 krb5kdc[26849](info): set up 2 sockets
Aug 07 22:56:06 dist-test-slave-z8x9 krb5kdc[26849](info): commencing operation
krb5kdc: starting...
W20260807 22:56:07.713858 25976 mini_kdc.cc:121] Time spent starting KDC: real 1.288s user 0.000s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786143361779964-25976-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786143361779964-25976-0/krb5kdc/test-user.keytab.
Aug 07 22:56:07 dist-test-slave-z8x9 krb5kdc[26849](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143367, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T22:56:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T22:56:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.94.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.62.keytab.
Entry for principal kudu/127.25.94.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.62.keytab.
I20260807 22:56:07.921517 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786143361779964-25976-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786143361779964-25976-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786143361779964-25976-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.62:41297
--webserver_interface=127.25.94.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.62.keytab
--principal=kudu/127.25.94.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.20:34989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.94.62:41297
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786143361779964-25976-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786143361779964-25976-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786143361779964-25976-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:56:08.266218 26867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:56:08.266547 26867 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:56:08.266639 26867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:56:08.274060 26867 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 22:56:08.274185 26867 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:56:08.274233 26867 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:56:08.274292 26867 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 22:56:08.274336 26867 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 22:56:08.284201 26867 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:34989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786143361779964-25976-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786143361779964-25976-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.94.62:41297
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.62.keytab
--principal=kudu/127.25.94.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.62:41297
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.94.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786143361779964-25976-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 22:56:08.285557 26867 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:56:08.286886 26867 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:56:08.298910 26877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:56:08.298934 26885 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:56:08.299000 26878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:56:08.299914 26867 server_base.cc:1034] running on GCE node
I20260807 22:56:08.300863 26867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:56:08.302547 26867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:56:08.303789 26867 hybrid_clock.cc:648] HybridClock initialized: now 1786143368303730 us; error 57 us; skew 500 ppm
I20260807 22:56:08.304162 26867 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 22:56:08 dist-test-slave-z8x9 krb5kdc[26849](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143368, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:56:08.311429 26867 init.cc:377] Logged in from keytab as kudu/127.25.94.62@KRBTEST.COM (short username kudu)
I20260807 22:56:08.313244 26867 webserver.cc:460] Webserver started at http://127.25.94.62:35781/ using document root <none> and password file <none>
I20260807 22:56:08.313997 26867 fs_manager.cc:359] Metadata directory not provided
I20260807 22:56:08.314145 26867 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:56:08.314477 26867 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:56:08.317618 26867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786143361779964-25976-0/minicluster-data/master-0/data/instance:
uuid: "b59d6be873dd47c1a907b43db20e707e"
format_stamp: "Formatted at 2026-08-07 22:56:08 on dist-test-slave-z8x9"
server_key: "d7cda5660ce88879b62a719b5a0dc7a9"
server_key_iv: "f6f28a4bd99d0b0234442d8a9c961b2f"
server_key_version: "encryptionkey@0"
I20260807 22:56:08.318392 26867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786143361779964-25976-0/minicluster-data/master-0/wal/instance:
uuid: "b59d6be873dd47c1a907b43db20e707e"
format_stamp: "Formatted at 2026-08-07 22:56:08 on dist-test-slave-z8x9"
server_key: "d7cda5660ce88879b62a719b5a0dc7a9"
server_key_iv: "f6f28a4bd99d0b0234442d8a9c961b2f"
server_key_version: "encryptionkey@0"
I20260807 22:56:08.324040 26867 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.005s
I20260807 22:56:08.327977 26897 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:56:08.329545 26867 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 22:56:08.329762 26867 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786143361779964-25976-0/minicluster-data/master-0/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786143361779964-25976-0/minicluster-data/master-0/wal
uuid: "b59d6be873dd47c1a907b43db20e707e"
format_stamp: "Formatted at 2026-08-07 22:56:08 on dist-test-slave-z8x9"
server_key: "d7cda5660ce88879b62a719b5a0dc7a9"
server_key_iv: "f6f28a4bd99d0b0234442d8a9c961b2f"
server_key_version: "encryptionkey@0"
I20260807 22:56:08.330011 26867 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786143361779964-25976-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786143361779964-25976-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786143361779964-25976-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:56:08.344403 26867 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:56:08.349041 26867 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:56:08.349412 26867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:56:08.369715 26867 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.62:41297
I20260807 22:56:08.369756 26975 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.62:41297 every 8 connection(s)
I20260807 22:56:08.371616 26867 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
I20260807 22:56:08.372277 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 26867
I20260807 22:56:08.372537 25976 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786143361779964-25976-0/minicluster-data/master-0/wal/instance
I20260807 22:56:08.373376 25976 external_mini_cluster.cc:1428] Setting key fde78f4c26c2a2539c005bb17027ed83
I20260807 22:56:08.376154 26976 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:56:08.387422 26976 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b59d6be873dd47c1a907b43db20e707e: Bootstrap starting.
Aug 07 22:56:08 dist-test-slave-z8x9 krb5kdc[26849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143367, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.62@KRBTEST.COM
I20260807 22:56:08.391326 26976 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b59d6be873dd47c1a907b43db20e707e: Neither blocks nor log segments found. Creating new log.
I20260807 22:56:08.392510 26976 log.cc:824] T 00000000000000000000000000000000 P b59d6be873dd47c1a907b43db20e707e: Log is configured to *not* fsync() on all Append() calls
I20260807 22:56:08.395750 26976 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b59d6be873dd47c1a907b43db20e707e: No bootstrap required, opened a new log
I20260807 22:56:08.397231 26979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:08.375364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46190 (local address 127.25.94.62:41297)
0807 22:56:08.376174 (+ 810us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:08.376190 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:08.376238 (+ 48us) server_negotiation.cc:407] Connection header received
0807 22:56:08.378269 (+ 2031us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:08.378293 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:08.378685 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:08.379097 (+ 412us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:56:08.380845 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:08.382276 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:08.383569 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:08.384301 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:08.384529 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:08.388555 (+ 4026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:56:08.388567 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:56:08.388582 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:56:08.388628 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:56:08.392494 (+ 3866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:08.393417 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:08.393424 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:08.393431 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:08.393740 (+ 309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:08.394201 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:08.394209 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:08.394213 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:08.394751 (+ 538us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:56:08.395075 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:08.395587 (+ 512us) server_negotiation.cc:299] Negotiation successful
0807 22:56:08.396101 (+ 514us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":600,"thread_start_us":397,"threads_started":1}
I20260807 22:56:08.404666 26976 raft_consensus.cc:348] T 00000000000000000000000000000000 P b59d6be873dd47c1a907b43db20e707e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b59d6be873dd47c1a907b43db20e707e" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 41297 } }
I20260807 22:56:08.405050 26976 raft_consensus.cc:374] T 00000000000000000000000000000000 P b59d6be873dd47c1a907b43db20e707e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:56:08.405221 26976 raft_consensus.cc:729] T 00000000000000000000000000000000 P b59d6be873dd47c1a907b43db20e707e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b59d6be873dd47c1a907b43db20e707e, State: Initialized, Role: FOLLOWER
I20260807 22:56:08.405778 26976 consensus_queue.cc:260] T 00000000000000000000000000000000 P b59d6be873dd47c1a907b43db20e707e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b59d6be873dd47c1a907b43db20e707e" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 41297 } }
I20260807 22:56:08.406031 26976 raft_consensus.cc:388] T 00000000000000000000000000000000 P b59d6be873dd47c1a907b43db20e707e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 22:56:08.406201 26976 raft_consensus.cc:482] T 00000000000000000000000000000000 P b59d6be873dd47c1a907b43db20e707e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 22:56:08.406378 26976 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b59d6be873dd47c1a907b43db20e707e [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:56:08.408703 26976 raft_consensus.cc:504] T 00000000000000000000000000000000 P b59d6be873dd47c1a907b43db20e707e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b59d6be873dd47c1a907b43db20e707e" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 41297 } }
I20260807 22:56:08.409345 26976 leader_election.cc:302] T 00000000000000000000000000000000 P b59d6be873dd47c1a907b43db20e707e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b59d6be873dd47c1a907b43db20e707e; no voters:
I20260807 22:56:08.409937 26976 leader_election.cc:288] T 00000000000000000000000000000000 P b59d6be873dd47c1a907b43db20e707e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 22:56:08.410090 26981 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b59d6be873dd47c1a907b43db20e707e [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:56:08.410845 26981 raft_consensus.cc:686] T 00000000000000000000000000000000 P b59d6be873dd47c1a907b43db20e707e [term 1 LEADER]: Becoming Leader. State: Replica: b59d6be873dd47c1a907b43db20e707e, State: Running, Role: LEADER
I20260807 22:56:08.411363 26981 consensus_queue.cc:237] T 00000000000000000000000000000000 P b59d6be873dd47c1a907b43db20e707e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b59d6be873dd47c1a907b43db20e707e" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 41297 } }
I20260807 22:56:08.412055 26976 sys_catalog.cc:564] T 00000000000000000000000000000000 P b59d6be873dd47c1a907b43db20e707e [sys.catalog]: configured and running, proceeding with master startup.
I20260807 22:56:08.414594 26982 sys_catalog.cc:455] T 00000000000000000000000000000000 P b59d6be873dd47c1a907b43db20e707e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b59d6be873dd47c1a907b43db20e707e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b59d6be873dd47c1a907b43db20e707e" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 41297 } } }
I20260807 22:56:08.415124 26982 sys_catalog.cc:458] T 00000000000000000000000000000000 P b59d6be873dd47c1a907b43db20e707e [sys.catalog]: This master's current role is: LEADER
I20260807 22:56:08.416072 26984 sys_catalog.cc:455] T 00000000000000000000000000000000 P b59d6be873dd47c1a907b43db20e707e [sys.catalog]: SysCatalogTable state changed. Reason: New leader b59d6be873dd47c1a907b43db20e707e. Latest consensus state: current_term: 1 leader_uuid: "b59d6be873dd47c1a907b43db20e707e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b59d6be873dd47c1a907b43db20e707e" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 41297 } } }
I20260807 22:56:08.416567 26984 sys_catalog.cc:458] T 00000000000000000000000000000000 P b59d6be873dd47c1a907b43db20e707e [sys.catalog]: This master's current role is: LEADER
I20260807 22:56:08.417457 26993 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 22:56:08.424829 26993 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 22:56:08.434368 26993 catalog_manager.cc:1347] Generated new cluster ID: 64237667b4574ac2ad4f9d4aca18f851
I20260807 22:56:08.434502 26993 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 22:56:08.456477 26993 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 22:56:08.457986 26993 catalog_manager.cc:1504] Loading token signing keys...
I20260807 22:56:08.475584 26993 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b59d6be873dd47c1a907b43db20e707e: Generated new TSK 0
I20260807 22:56:08.476922 26993 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.94.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.1.keytab.
Entry for principal kudu/127.25.94.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.1.keytab.
I20260807 22:56:08.536635 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786143361779964-25976-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786143361779964-25976-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786143361779964-25976-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786143361779964-25976-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.1:0
--local_ip_for_outbound_sockets=127.25.94.1
--webserver_interface=127.25.94.1
--webserver_port=0
--tserver_master_addrs=127.25.94.62:41297
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.1.keytab
--principal=kudu/127.25.94.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.20:34989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786143361779964-25976-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786143361779964-25976-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786143361779964-25976-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:56:08.881412 27011 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:56:08.881700 27011 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:56:08.881803 27011 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:56:08.888907 27011 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:56:08.889021 27011 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:56:08.889138 27011 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.1
I20260807 22:56:08.899072 27011 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:34989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786143361779964-25976-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786143361779964-25976-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.1.keytab
--principal=kudu/127.25.94.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786143361779964-25976-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.94.1
--webserver_port=0
--tserver_master_addrs=127.25.94.62:41297
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.1
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786143361779964-25976-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 22:56:08.900296 27011 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:56:08.901631 27011 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:56:08.915082 27021 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:56:08.915119 27019 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:56:08.915376 27017 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:56:08.915894 27011 server_base.cc:1034] running on GCE node
I20260807 22:56:08.916666 27011 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:56:08.917764 27011 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:56:08.919010 27011 hybrid_clock.cc:648] HybridClock initialized: now 1786143368918967 us; error 38 us; skew 500 ppm
I20260807 22:56:08.919371 27011 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 22:56:08 dist-test-slave-z8x9 krb5kdc[26849](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143368, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:56:08.922761 27011 init.cc:377] Logged in from keytab as kudu/127.25.94.1@KRBTEST.COM (short username kudu)
I20260807 22:56:08.924198 27011 webserver.cc:460] Webserver started at http://127.25.94.1:37005/ using document root <none> and password file <none>
I20260807 22:56:08.924892 27011 fs_manager.cc:359] Metadata directory not provided
I20260807 22:56:08.924993 27011 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:56:08.925441 27011 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:56:08.928421 27011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786143361779964-25976-0/minicluster-data/ts-0/data/instance:
uuid: "059258caec4e43a9858bde7b62fb3ad2"
format_stamp: "Formatted at 2026-08-07 22:56:08 on dist-test-slave-z8x9"
server_key: "ece05730b70a16914812a3ccabe56885"
server_key_iv: "153a5d0b4ff94b42455990bc1cd8fd32"
server_key_version: "encryptionkey@0"
I20260807 22:56:08.929312 27011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786143361779964-25976-0/minicluster-data/ts-0/wal/instance:
uuid: "059258caec4e43a9858bde7b62fb3ad2"
format_stamp: "Formatted at 2026-08-07 22:56:08 on dist-test-slave-z8x9"
server_key: "ece05730b70a16914812a3ccabe56885"
server_key_iv: "153a5d0b4ff94b42455990bc1cd8fd32"
server_key_version: "encryptionkey@0"
I20260807 22:56:08.934743 27011 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260807 22:56:08.938472 27032 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:56:08.939855 27011 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 22:56:08.940071 27011 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786143361779964-25976-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786143361779964-25976-0/minicluster-data/ts-0/wal
uuid: "059258caec4e43a9858bde7b62fb3ad2"
format_stamp: "Formatted at 2026-08-07 22:56:08 on dist-test-slave-z8x9"
server_key: "ece05730b70a16914812a3ccabe56885"
server_key_iv: "153a5d0b4ff94b42455990bc1cd8fd32"
server_key_version: "encryptionkey@0"
I20260807 22:56:08.940291 27011 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786143361779964-25976-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786143361779964-25976-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786143361779964-25976-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:56:08.961450 27011 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:56:08.966393 27011 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:56:08.966717 27011 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:56:08.967761 27011 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:56:08.969213 27011 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:56:08.969309 27011 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:56:08.969451 27011 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:56:08.969514 27011 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:56:09.004822 27011 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.1:34021
I20260807 22:56:09.004936 27197 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.1:34021 every 8 connection(s)
I20260807 22:56:09.006196 27011 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786143361779964-25976-0/minicluster-data/ts-0/data/info.pb
I20260807 22:56:09.007364 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 27011
I20260807 22:56:09.007625 25976 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786143361779964-25976-0/minicluster-data/ts-0/wal/instance
I20260807 22:56:09.008463 25976 external_mini_cluster.cc:1428] Setting key c6ca7d1a9d203cbb623889e681cf42af
Aug 07 22:56:09 dist-test-slave-z8x9 krb5kdc[26849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143368, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.1@KRBTEST.COM for kudu/127.25.94.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.25.94.2@KRBTEST.COM; defaulting to no policy
I20260807 22:56:09.030869 27201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:09.009684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.1:41849 (local address 127.25.94.62:41297)
0807 22:56:09.010199 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:09.010207 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:09.011126 (+ 919us) server_negotiation.cc:407] Connection header received
0807 22:56:09.014231 (+ 3105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:09.014236 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:09.014304 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:09.014400 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:56:09.016551 (+ 2151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:09.017357 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:09.019950 (+ 2593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:09.020340 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:09.020439 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:09.024832 (+ 4393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:56:09.024838 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:56:09.024841 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:56:09.024875 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:56:09.027626 (+ 2751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:09.028429 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:09.028436 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:09.028440 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:09.028573 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:09.029020 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:09.029025 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:09.029029 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:09.029379 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:56:09.029547 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:09.030237 (+ 690us) server_negotiation.cc:299] Negotiation successful
0807 22:56:09.030473 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":254,"threads_started":1}
I20260807 22:56:09.031991 27203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:09.010000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:41297 (local address 127.25.94.1:41849)
0807 22:56:09.010934 (+ 934us) negotiation.cc:107] Waiting for socket to connect
0807 22:56:09.010996 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0807 22:56:09.013925 (+ 2929us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:56:09.014683 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:56:09.014697 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:56:09.015517 (+ 820us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:56:09.016363 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:09.016392 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:09.017522 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:09.017530 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:09.019768 (+ 2238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:09.019797 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:09.020579 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:09.020589 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:09.020935 (+ 346us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:09.021737 (+ 802us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:56:09.021776 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:56:09.024592 (+ 2816us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:56:09.027784 (+ 3192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:56:09.027793 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:56:09.027811 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:56:09.028270 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:56:09.028693 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:56:09.028699 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:56:09.028702 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:56:09.028904 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:56:09.029502 (+ 598us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:56:09.029512 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:56:09.029875 (+ 363us) client_negotiation.cc:769] Sending connection context
0807 22:56:09.030149 (+ 274us) client_negotiation.cc:240] Negotiation successful
0807 22:56:09.030751 (+ 602us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":626,"thread_start_us":382,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.2@KRBTEST.COM" created.
I20260807 22:56:09.033795 27198 heartbeater.cc:347] Connected to a master server at 127.25.94.62:41297
I20260807 22:56:09.034147 27198 heartbeater.cc:464] Registering TS with master...
I20260807 22:56:09.035100 27198 heartbeater.cc:511] Master 127.25.94.62:41297 requested a full tablet report, sending...
I20260807 22:56:09.037685 26924 ts_manager.cc:194] Registered new tserver with Master: 059258caec4e43a9858bde7b62fb3ad2 (127.25.94.1:34021)
I20260807 22:56:09.039201 26924 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.94.1@KRBTEST.COM'} at 127.25.94.1:41849
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.2.keytab.
Entry for principal kudu/127.25.94.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.2.keytab.
I20260807 22:56:09.056892 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786143361779964-25976-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786143361779964-25976-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786143361779964-25976-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786143361779964-25976-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.2:0
--local_ip_for_outbound_sockets=127.25.94.2
--webserver_interface=127.25.94.2
--webserver_port=0
--tserver_master_addrs=127.25.94.62:41297
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.2.keytab
--principal=kudu/127.25.94.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.20:34989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786143361779964-25976-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786143361779964-25976-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786143361779964-25976-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:56:09.409593 27208 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:56:09.409915 27208 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:56:09.410099 27208 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:56:09.417822 27208 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:56:09.417937 27208 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:56:09.418157 27208 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.2
I20260807 22:56:09.428407 27208 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:34989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786143361779964-25976-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786143361779964-25976-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.2.keytab
--principal=kudu/127.25.94.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786143361779964-25976-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.94.2
--webserver_port=0
--tserver_master_addrs=127.25.94.62:41297
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.2
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786143361779964-25976-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 22:56:09.429857 27208 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:56:09.431281 27208 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:56:09.445726 27222 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:56:09.445792 27216 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:56:09.445758 27217 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:56:09.447037 27208 server_base.cc:1034] running on GCE node
I20260807 22:56:09.447767 27208 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:56:09.449101 27208 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:56:09.450366 27208 hybrid_clock.cc:648] HybridClock initialized: now 1786143369450265 us; error 109 us; skew 500 ppm
I20260807 22:56:09.450799 27208 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 22:56:09 dist-test-slave-z8x9 krb5kdc[26849](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143369, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:56:09.454584 27208 init.cc:377] Logged in from keytab as kudu/127.25.94.2@KRBTEST.COM (short username kudu)
I20260807 22:56:09.456101 27208 webserver.cc:460] Webserver started at http://127.25.94.2:37397/ using document root <none> and password file <none>
I20260807 22:56:09.456835 27208 fs_manager.cc:359] Metadata directory not provided
I20260807 22:56:09.456974 27208 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:56:09.457397 27208 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:56:09.460414 27208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786143361779964-25976-0/minicluster-data/ts-1/data/instance:
uuid: "4e6df4a857944cee964d912a0760c0da"
format_stamp: "Formatted at 2026-08-07 22:56:09 on dist-test-slave-z8x9"
server_key: "2850bd12a2c99743c9b5b6119b2a5664"
server_key_iv: "b7a22f2fb8ae774d96ae02c78b752463"
server_key_version: "encryptionkey@0"
I20260807 22:56:09.461288 27208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786143361779964-25976-0/minicluster-data/ts-1/wal/instance:
uuid: "4e6df4a857944cee964d912a0760c0da"
format_stamp: "Formatted at 2026-08-07 22:56:09 on dist-test-slave-z8x9"
server_key: "2850bd12a2c99743c9b5b6119b2a5664"
server_key_iv: "b7a22f2fb8ae774d96ae02c78b752463"
server_key_version: "encryptionkey@0"
I20260807 22:56:09.466820 27208 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260807 22:56:09.470506 27239 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:56:09.471858 27208 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 22:56:09.472061 27208 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786143361779964-25976-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786143361779964-25976-0/minicluster-data/ts-1/wal
uuid: "4e6df4a857944cee964d912a0760c0da"
format_stamp: "Formatted at 2026-08-07 22:56:09 on dist-test-slave-z8x9"
server_key: "2850bd12a2c99743c9b5b6119b2a5664"
server_key_iv: "b7a22f2fb8ae774d96ae02c78b752463"
server_key_version: "encryptionkey@0"
I20260807 22:56:09.472270 27208 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786143361779964-25976-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786143361779964-25976-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786143361779964-25976-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:56:09.500550 27208 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:56:09.504952 27208 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:56:09.505302 27208 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:56:09.506320 27208 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:56:09.507665 27208 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:56:09.507740 27208 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:56:09.507916 27208 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:56:09.507969 27208 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:56:09.544219 27208 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.2:35891
I20260807 22:56:09.544358 27404 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.2:35891 every 8 connection(s)
I20260807 22:56:09.545686 27208 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786143361779964-25976-0/minicluster-data/ts-1/data/info.pb
I20260807 22:56:09.547312 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 27208
I20260807 22:56:09.547526 25976 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786143361779964-25976-0/minicluster-data/ts-1/wal/instance
I20260807 22:56:09.548134 25976 external_mini_cluster.cc:1428] Setting key 027a973888e3bd69e39f9c3bb1007c4e
Aug 07 22:56:09 dist-test-slave-z8x9 krb5kdc[26849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143369, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.2@KRBTEST.COM for kudu/127.25.94.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.25.94.3@KRBTEST.COM; defaulting to no policy
I20260807 22:56:09.570261 27407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:09.548661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.2:32951 (local address 127.25.94.62:41297)
0807 22:56:09.549226 (+ 565us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:09.549233 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:09.550245 (+ 1012us) server_negotiation.cc:407] Connection header received
0807 22:56:09.553291 (+ 3046us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:09.553297 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:09.553377 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:09.553472 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:56:09.555740 (+ 2268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:09.556504 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:09.558219 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:09.559014 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:09.559162 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:09.562836 (+ 3674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:56:09.562846 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:56:09.562854 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:56:09.562903 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:56:09.566945 (+ 4042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:09.567822 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:09.567828 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:09.567834 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:09.568001 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:09.568454 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:09.568461 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:09.568464 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:09.568866 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:56:09.569032 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:09.569812 (+ 780us) server_negotiation.cc:299] Negotiation successful
0807 22:56:09.569982 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":277,"threads_started":1}
I20260807 22:56:09.571241 27408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:09.549294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:41297 (local address 127.25.94.2:32951)
0807 22:56:09.550083 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0807 22:56:09.550143 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0807 22:56:09.552863 (+ 2720us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:56:09.553768 (+ 905us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:56:09.553783 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:56:09.554617 (+ 834us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:56:09.555515 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:09.555545 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:09.556714 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:09.556718 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:09.558056 (+ 1338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:09.558069 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:09.559244 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:09.559249 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:09.559473 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:09.560214 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:56:09.560239 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:56:09.562549 (+ 2310us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:56:09.567128 (+ 4579us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:56:09.567138 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:56:09.567154 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:56:09.567665 (+ 511us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:56:09.568117 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:56:09.568120 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:56:09.568122 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:56:09.568321 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:56:09.568992 (+ 671us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:56:09.568998 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:56:09.569388 (+ 390us) client_negotiation.cc:769] Sending connection context
0807 22:56:09.569679 (+ 291us) client_negotiation.cc:240] Negotiation successful
0807 22:56:09.570097 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":328,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.3@KRBTEST.COM" created.
I20260807 22:56:09.572876 27405 heartbeater.cc:347] Connected to a master server at 127.25.94.62:41297
I20260807 22:56:09.573240 27405 heartbeater.cc:464] Registering TS with master...
I20260807 22:56:09.573971 27405 heartbeater.cc:511] Master 127.25.94.62:41297 requested a full tablet report, sending...
I20260807 22:56:09.575583 26924 ts_manager.cc:194] Registered new tserver with Master: 4e6df4a857944cee964d912a0760c0da (127.25.94.2:35891)
I20260807 22:56:09.576853 26924 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.94.2@KRBTEST.COM'} at 127.25.94.2:32951
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.3.keytab.
Entry for principal kudu/127.25.94.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.3.keytab.
I20260807 22:56:09.597756 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786143361779964-25976-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786143361779964-25976-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786143361779964-25976-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786143361779964-25976-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.3:0
--local_ip_for_outbound_sockets=127.25.94.3
--webserver_interface=127.25.94.3
--webserver_port=0
--tserver_master_addrs=127.25.94.62:41297
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.3.keytab
--principal=kudu/127.25.94.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.20:34989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786143361779964-25976-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786143361779964-25976-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786143361779964-25976-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:56:09.930608 27412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:56:09.930936 27412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:56:09.931107 27412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:56:09.938572 27412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:56:09.938725 27412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:56:09.938903 27412 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.3
I20260807 22:56:09.948985 27412 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:34989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786143361779964-25976-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786143361779964-25976-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.3.keytab
--principal=kudu/127.25.94.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786143361779964-25976-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.94.3
--webserver_port=0
--tserver_master_addrs=127.25.94.62:41297
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.3
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786143361779964-25976-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 22:56:09.950380 27412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:56:09.951754 27412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:56:09.965353 27420 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:56:09.965399 27423 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:56:09.965444 27425 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:56:09.966274 27412 server_base.cc:1034] running on GCE node
I20260807 22:56:09.966990 27412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:56:09.967981 27412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:56:09.969280 27412 hybrid_clock.cc:648] HybridClock initialized: now 1786143369969220 us; error 53 us; skew 500 ppm
I20260807 22:56:09.969650 27412 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 22:56:09 dist-test-slave-z8x9 krb5kdc[26849](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143369, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:56:09.973304 27412 init.cc:377] Logged in from keytab as kudu/127.25.94.3@KRBTEST.COM (short username kudu)
I20260807 22:56:09.974836 27412 webserver.cc:460] Webserver started at http://127.25.94.3:33653/ using document root <none> and password file <none>
I20260807 22:56:09.975569 27412 fs_manager.cc:359] Metadata directory not provided
I20260807 22:56:09.975742 27412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:56:09.976095 27412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:56:09.979245 27412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786143361779964-25976-0/minicluster-data/ts-2/data/instance:
uuid: "0c2426a5b37b4aaebafeeb99445d886d"
format_stamp: "Formatted at 2026-08-07 22:56:09 on dist-test-slave-z8x9"
server_key: "8ffca3da11d2293e946575de5aec5d3b"
server_key_iv: "d6994f141d422957d0a1d5d4f08b4baf"
server_key_version: "encryptionkey@0"
I20260807 22:56:09.980005 27412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786143361779964-25976-0/minicluster-data/ts-2/wal/instance:
uuid: "0c2426a5b37b4aaebafeeb99445d886d"
format_stamp: "Formatted at 2026-08-07 22:56:09 on dist-test-slave-z8x9"
server_key: "8ffca3da11d2293e946575de5aec5d3b"
server_key_iv: "d6994f141d422957d0a1d5d4f08b4baf"
server_key_version: "encryptionkey@0"
I20260807 22:56:09.985425 27412 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260807 22:56:09.988813 27440 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:56:09.990183 27412 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 22:56:09.990327 27412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786143361779964-25976-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786143361779964-25976-0/minicluster-data/ts-2/wal
uuid: "0c2426a5b37b4aaebafeeb99445d886d"
format_stamp: "Formatted at 2026-08-07 22:56:09 on dist-test-slave-z8x9"
server_key: "8ffca3da11d2293e946575de5aec5d3b"
server_key_iv: "d6994f141d422957d0a1d5d4f08b4baf"
server_key_version: "encryptionkey@0"
I20260807 22:56:09.990494 27412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786143361779964-25976-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786143361779964-25976-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786143361779964-25976-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:56:10.005761 27412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:56:10.010197 27412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:56:10.010507 27412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:56:10.011466 27412 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:56:10.013059 27412 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:56:10.013136 27412 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:56:10.013286 27412 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:56:10.013341 27412 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:56:10.042591 27198 heartbeater.cc:503] Master 127.25.94.62:41297 was elected leader, sending a full tablet report...
I20260807 22:56:10.049456 27412 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.3:39579
I20260807 22:56:10.049489 27590 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.3:39579 every 8 connection(s)
I20260807 22:56:10.050788 27412 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786143361779964-25976-0/minicluster-data/ts-2/data/info.pb
I20260807 22:56:10.057472 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 27412
I20260807 22:56:10.057693 25976 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786143361779964-25976-0/minicluster-data/ts-2/wal/instance
I20260807 22:56:10.058130 25976 external_mini_cluster.cc:1428] Setting key a5d689f03bf80314be4f5ff470c67711
Aug 07 22:56:10 dist-test-slave-z8x9 krb5kdc[26849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143369, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.3@KRBTEST.COM for kudu/127.25.94.62@KRBTEST.COM
I20260807 22:56:10.072134 27407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:10.053584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.3:48123 (local address 127.25.94.62:41297)
0807 22:56:10.053806 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:10.053813 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:10.054898 (+ 1085us) server_negotiation.cc:407] Connection header received
0807 22:56:10.056771 (+ 1873us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:10.056776 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:10.056866 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:10.056960 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:56:10.058747 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:10.059543 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:10.061101 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:10.061574 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:10.061714 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:10.065420 (+ 3706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:56:10.065425 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:56:10.065429 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:56:10.065465 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:56:10.068730 (+ 3265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:10.069643 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:10.069647 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:10.069649 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:10.069781 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:10.070261 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:10.070265 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:10.070267 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:10.070603 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:56:10.070764 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:10.071488 (+ 724us) server_negotiation.cc:299] Negotiation successful
0807 22:56:10.071723 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260807 22:56:10.073383 27597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:10.053990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:41297 (local address 127.25.94.3:48123)
0807 22:56:10.054721 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0807 22:56:10.054775 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0807 22:56:10.056503 (+ 1728us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:56:10.057257 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:56:10.057269 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:56:10.057911 (+ 642us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:56:10.058551 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:10.058568 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:10.059720 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:10.059724 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:10.060953 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:10.060965 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:10.061837 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:10.061845 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:10.062088 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:10.062740 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:56:10.062778 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:56:10.065099 (+ 2321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:56:10.068923 (+ 3824us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:56:10.068940 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:56:10.068955 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:56:10.069498 (+ 543us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:56:10.069908 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:56:10.069913 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:56:10.069924 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:56:10.070129 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:56:10.070742 (+ 613us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:56:10.070750 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:56:10.071108 (+ 358us) client_negotiation.cc:769] Sending connection context
0807 22:56:10.071383 (+ 275us) client_negotiation.cc:240] Negotiation successful
0807 22:56:10.071975 (+ 592us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":285,"threads_started":1}
I20260807 22:56:10.074779 27591 heartbeater.cc:347] Connected to a master server at 127.25.94.62:41297
I20260807 22:56:10.075093 27591 heartbeater.cc:464] Registering TS with master...
I20260807 22:56:10.075834 27591 heartbeater.cc:511] Master 127.25.94.62:41297 requested a full tablet report, sending...
I20260807 22:56:10.077378 26924 ts_manager.cc:194] Registered new tserver with Master: 0c2426a5b37b4aaebafeeb99445d886d (127.25.94.3:39579)
I20260807 22:56:10.078210 26924 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.94.3@KRBTEST.COM'} at 127.25.94.3:48123
I20260807 22:56:10.087733 25976 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 22:56:10.113641 27407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:10.096574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46206 (local address 127.25.94.62:41297)
0807 22:56:10.096825 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:10.096834 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:10.097469 (+ 635us) server_negotiation.cc:407] Connection header received
0807 22:56:10.097646 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:10.097651 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:10.097732 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:10.097843 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:56:10.099952 (+ 2109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:10.100715 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:10.102560 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:10.103232 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:10.103340 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:10.106092 (+ 2752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:56:10.106097 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:56:10.106101 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:56:10.106136 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:56:10.109744 (+ 3608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:10.110692 (+ 948us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:10.110698 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:10.110700 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:10.111020 (+ 320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:10.111534 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:10.111544 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:10.111547 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:10.112082 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:56:10.112458 (+ 376us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:10.112945 (+ 487us) server_negotiation.cc:299] Negotiation successful
0807 22:56:10.113221 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260807 22:56:10.123387 26924 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46206:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 22:56:10.132186 26924 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 22:56:10.180737 27615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:10.151158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.1:34021 (local address 127.0.0.1:38462)
0807 22:56:10.163377 (+ 12219us) negotiation.cc:107] Waiting for socket to connect
0807 22:56:10.163444 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0807 22:56:10.163824 (+ 380us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:56:10.170798 (+ 6974us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:56:10.170809 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:56:10.170867 (+ 58us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:56:10.171435 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:10.171456 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:10.174578 (+ 3122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:10.174586 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:10.177381 (+ 2795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:10.177399 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:10.179500 (+ 2101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:10.179507 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:10.179751 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:10.179811 (+ 60us) client_negotiation.cc:769] Sending connection context
0807 22:56:10.179948 (+ 137us) client_negotiation.cc:240] Negotiation successful
0807 22:56:10.180182 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11833,"mutex_wait_us":47,"thread_start_us":15281,"threads_started":1}
I20260807 22:56:10.181596 27616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:10.151355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38462 (local address 127.25.94.1:34021)
0807 22:56:10.168053 (+ 16698us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:10.168069 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:10.168117 (+ 48us) server_negotiation.cc:407] Connection header received
0807 22:56:10.168260 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:10.168273 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:10.170438 (+ 2165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:10.171715 (+ 1277us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:56:10.172075 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:10.174387 (+ 2312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:10.177581 (+ 3194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:10.179249 (+ 1668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:10.180607 (+ 1358us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:10.180919 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:10.181006 (+ 87us) server_negotiation.cc:299] Negotiation successful
0807 22:56:10.181248 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16406,"thread_start_us":16181,"threads_started":1}
I20260807 22:56:10.182117 27618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:10.151717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.3:39579 (local address 127.0.0.1:45042)
0807 22:56:10.171622 (+ 19905us) negotiation.cc:107] Waiting for socket to connect
0807 22:56:10.171657 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0807 22:56:10.171871 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:56:10.173241 (+ 1370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:56:10.173248 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:56:10.173285 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:56:10.173728 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:10.173744 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:10.175779 (+ 2035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:10.175787 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:10.178044 (+ 2257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:10.178057 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:10.179913 (+ 1856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:10.179920 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:10.180113 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:10.180161 (+ 48us) client_negotiation.cc:769] Sending connection context
0807 22:56:10.180543 (+ 382us) client_negotiation.cc:240] Negotiation successful
0807 22:56:10.180781 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19712,"mutex_wait_us":1,"thread_start_us":20376,"threads_started":1}
I20260807 22:56:10.182969 27622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:10.152520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45042 (local address 127.25.94.3:39579)
0807 22:56:10.170643 (+ 18123us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:10.170655 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:10.172088 (+ 1433us) server_negotiation.cc:407] Connection header received
0807 22:56:10.172261 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:10.172273 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:10.172604 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:10.172850 (+ 246us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:56:10.173984 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:10.175646 (+ 1662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:10.178224 (+ 2578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:10.179362 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:10.179497 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:10.179749 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:10.181419 (+ 1670us) server_negotiation.cc:299] Negotiation successful
0807 22:56:10.181598 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17839,"thread_start_us":18147,"threads_started":1}
I20260807 22:56:10.183120 27614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:10.151259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.2:35891 (local address 127.0.0.1:44068)
0807 22:56:10.166798 (+ 15539us) negotiation.cc:107] Waiting for socket to connect
0807 22:56:10.166827 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0807 22:56:10.167017 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:56:10.173952 (+ 6935us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:56:10.173957 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:56:10.173989 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:56:10.174547 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:10.174563 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:10.176705 (+ 2142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:10.176710 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:10.178848 (+ 2138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:10.178865 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:10.181533 (+ 2668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:10.181557 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:10.182095 (+ 538us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:10.182179 (+ 84us) client_negotiation.cc:769] Sending connection context
0807 22:56:10.182328 (+ 149us) client_negotiation.cc:240] Negotiation successful
0807 22:56:10.182705 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15345,"thread_start_us":12865,"threads_started":1}
I20260807 22:56:10.184276 27621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:10.151841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44068 (local address 127.25.94.2:35891)
0807 22:56:10.172907 (+ 21066us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:10.172917 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:10.172959 (+ 42us) server_negotiation.cc:407] Connection header received
0807 22:56:10.173100 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:10.173118 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:10.173664 (+ 546us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:10.174770 (+ 1106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:56:10.175011 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:10.176553 (+ 1542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:10.179068 (+ 2515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:10.181231 (+ 2163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:10.183334 (+ 2103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:10.183648 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:10.183734 (+ 86us) server_negotiation.cc:299] Negotiation successful
0807 22:56:10.183922 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20835,"thread_start_us":21843,"threads_started":1}
I20260807 22:56:10.187252 27109 tablet_service.cc:1467] Processing CreateTablet for tablet 3f18f698f43f42cca80a60bf1bb5de42 (DEFAULT_TABLE table=test-table [id=340dd06dad6f4a46813c780b94bb8f86]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:56:10.189370 27109 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f18f698f43f42cca80a60bf1bb5de42. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:56:10.188782 27314 tablet_service.cc:1467] Processing CreateTablet for tablet 3f18f698f43f42cca80a60bf1bb5de42 (DEFAULT_TABLE table=test-table [id=340dd06dad6f4a46813c780b94bb8f86]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:56:10.190372 27314 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f18f698f43f42cca80a60bf1bb5de42. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:56:10.191224 27509 tablet_service.cc:1467] Processing CreateTablet for tablet 3f18f698f43f42cca80a60bf1bb5de42 (DEFAULT_TABLE table=test-table [id=340dd06dad6f4a46813c780b94bb8f86]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:56:10.193763 27509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f18f698f43f42cca80a60bf1bb5de42. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:56:10.208870 27625 tablet_bootstrap.cc:492] T 3f18f698f43f42cca80a60bf1bb5de42 P 4e6df4a857944cee964d912a0760c0da: Bootstrap starting.
I20260807 22:56:10.210935 27626 tablet_bootstrap.cc:492] T 3f18f698f43f42cca80a60bf1bb5de42 P 059258caec4e43a9858bde7b62fb3ad2: Bootstrap starting.
I20260807 22:56:10.213568 27627 tablet_bootstrap.cc:492] T 3f18f698f43f42cca80a60bf1bb5de42 P 0c2426a5b37b4aaebafeeb99445d886d: Bootstrap starting.
I20260807 22:56:10.216091 27625 tablet_bootstrap.cc:654] T 3f18f698f43f42cca80a60bf1bb5de42 P 4e6df4a857944cee964d912a0760c0da: Neither blocks nor log segments found. Creating new log.
I20260807 22:56:10.217036 27626 tablet_bootstrap.cc:654] T 3f18f698f43f42cca80a60bf1bb5de42 P 059258caec4e43a9858bde7b62fb3ad2: Neither blocks nor log segments found. Creating new log.
I20260807 22:56:10.218072 27625 log.cc:824] T 3f18f698f43f42cca80a60bf1bb5de42 P 4e6df4a857944cee964d912a0760c0da: Log is configured to *not* fsync() on all Append() calls
I20260807 22:56:10.218971 27627 tablet_bootstrap.cc:654] T 3f18f698f43f42cca80a60bf1bb5de42 P 0c2426a5b37b4aaebafeeb99445d886d: Neither blocks nor log segments found. Creating new log.
I20260807 22:56:10.219434 27626 log.cc:824] T 3f18f698f43f42cca80a60bf1bb5de42 P 059258caec4e43a9858bde7b62fb3ad2: Log is configured to *not* fsync() on all Append() calls
I20260807 22:56:10.220628 27627 log.cc:824] T 3f18f698f43f42cca80a60bf1bb5de42 P 0c2426a5b37b4aaebafeeb99445d886d: Log is configured to *not* fsync() on all Append() calls
I20260807 22:56:10.222916 27625 tablet_bootstrap.cc:492] T 3f18f698f43f42cca80a60bf1bb5de42 P 4e6df4a857944cee964d912a0760c0da: No bootstrap required, opened a new log
I20260807 22:56:10.223260 27625 ts_tablet_manager.cc:1397] T 3f18f698f43f42cca80a60bf1bb5de42 P 4e6df4a857944cee964d912a0760c0da: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.003s
I20260807 22:56:10.226315 27627 tablet_bootstrap.cc:492] T 3f18f698f43f42cca80a60bf1bb5de42 P 0c2426a5b37b4aaebafeeb99445d886d: No bootstrap required, opened a new log
I20260807 22:56:10.226475 27626 tablet_bootstrap.cc:492] T 3f18f698f43f42cca80a60bf1bb5de42 P 059258caec4e43a9858bde7b62fb3ad2: No bootstrap required, opened a new log
I20260807 22:56:10.226787 27626 ts_tablet_manager.cc:1397] T 3f18f698f43f42cca80a60bf1bb5de42 P 059258caec4e43a9858bde7b62fb3ad2: Time spent bootstrapping tablet: real 0.016s user 0.014s sys 0.000s
I20260807 22:56:10.226927 27627 ts_tablet_manager.cc:1397] T 3f18f698f43f42cca80a60bf1bb5de42 P 0c2426a5b37b4aaebafeeb99445d886d: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.005s
I20260807 22:56:10.234658 27627 raft_consensus.cc:348] T 3f18f698f43f42cca80a60bf1bb5de42 P 0c2426a5b37b4aaebafeeb99445d886d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059258caec4e43a9858bde7b62fb3ad2" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 34021 } } peers { permanent_uuid: "4e6df4a857944cee964d912a0760c0da" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 35891 } } peers { permanent_uuid: "0c2426a5b37b4aaebafeeb99445d886d" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 39579 } }
I20260807 22:56:10.234829 27625 raft_consensus.cc:348] T 3f18f698f43f42cca80a60bf1bb5de42 P 4e6df4a857944cee964d912a0760c0da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059258caec4e43a9858bde7b62fb3ad2" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 34021 } } peers { permanent_uuid: "4e6df4a857944cee964d912a0760c0da" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 35891 } } peers { permanent_uuid: "0c2426a5b37b4aaebafeeb99445d886d" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 39579 } }
I20260807 22:56:10.235152 27627 raft_consensus.cc:374] T 3f18f698f43f42cca80a60bf1bb5de42 P 0c2426a5b37b4aaebafeeb99445d886d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:56:10.235255 27627 raft_consensus.cc:729] T 3f18f698f43f42cca80a60bf1bb5de42 P 0c2426a5b37b4aaebafeeb99445d886d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c2426a5b37b4aaebafeeb99445d886d, State: Initialized, Role: FOLLOWER
I20260807 22:56:10.235280 27625 raft_consensus.cc:374] T 3f18f698f43f42cca80a60bf1bb5de42 P 4e6df4a857944cee964d912a0760c0da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:56:10.235378 27625 raft_consensus.cc:729] T 3f18f698f43f42cca80a60bf1bb5de42 P 4e6df4a857944cee964d912a0760c0da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e6df4a857944cee964d912a0760c0da, State: Initialized, Role: FOLLOWER
I20260807 22:56:10.236020 27627 consensus_queue.cc:260] T 3f18f698f43f42cca80a60bf1bb5de42 P 0c2426a5b37b4aaebafeeb99445d886d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059258caec4e43a9858bde7b62fb3ad2" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 34021 } } peers { permanent_uuid: "4e6df4a857944cee964d912a0760c0da" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 35891 } } peers { permanent_uuid: "0c2426a5b37b4aaebafeeb99445d886d" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 39579 } }
I20260807 22:56:10.236263 27625 consensus_queue.cc:260] T 3f18f698f43f42cca80a60bf1bb5de42 P 4e6df4a857944cee964d912a0760c0da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059258caec4e43a9858bde7b62fb3ad2" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 34021 } } peers { permanent_uuid: "4e6df4a857944cee964d912a0760c0da" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 35891 } } peers { permanent_uuid: "0c2426a5b37b4aaebafeeb99445d886d" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 39579 } }
I20260807 22:56:10.238276 27626 raft_consensus.cc:348] T 3f18f698f43f42cca80a60bf1bb5de42 P 059258caec4e43a9858bde7b62fb3ad2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059258caec4e43a9858bde7b62fb3ad2" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 34021 } } peers { permanent_uuid: "4e6df4a857944cee964d912a0760c0da" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 35891 } } peers { permanent_uuid: "0c2426a5b37b4aaebafeeb99445d886d" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 39579 } }
I20260807 22:56:10.238749 27626 raft_consensus.cc:374] T 3f18f698f43f42cca80a60bf1bb5de42 P 059258caec4e43a9858bde7b62fb3ad2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:56:10.238755 27405 heartbeater.cc:503] Master 127.25.94.62:41297 was elected leader, sending a full tablet report...
I20260807 22:56:10.238849 27626 raft_consensus.cc:729] T 3f18f698f43f42cca80a60bf1bb5de42 P 059258caec4e43a9858bde7b62fb3ad2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059258caec4e43a9858bde7b62fb3ad2, State: Initialized, Role: FOLLOWER
I20260807 22:56:10.239382 27625 ts_tablet_manager.cc:1428] T 3f18f698f43f42cca80a60bf1bb5de42 P 4e6df4a857944cee964d912a0760c0da: Time spent starting tablet: real 0.016s user 0.013s sys 0.003s
I20260807 22:56:10.239487 27626 consensus_queue.cc:260] T 3f18f698f43f42cca80a60bf1bb5de42 P 059258caec4e43a9858bde7b62fb3ad2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059258caec4e43a9858bde7b62fb3ad2" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 34021 } } peers { permanent_uuid: "4e6df4a857944cee964d912a0760c0da" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 35891 } } peers { permanent_uuid: "0c2426a5b37b4aaebafeeb99445d886d" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 39579 } }
I20260807 22:56:10.241950 27626 ts_tablet_manager.cc:1428] T 3f18f698f43f42cca80a60bf1bb5de42 P 059258caec4e43a9858bde7b62fb3ad2: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
I20260807 22:56:10.246176 27591 heartbeater.cc:503] Master 127.25.94.62:41297 was elected leader, sending a full tablet report...
I20260807 22:56:10.247092 27627 ts_tablet_manager.cc:1428] T 3f18f698f43f42cca80a60bf1bb5de42 P 0c2426a5b37b4aaebafeeb99445d886d: Time spent starting tablet: real 0.020s user 0.013s sys 0.007s
W20260807 22:56:10.259027 27199 tablet.cc:2369] T 3f18f698f43f42cca80a60bf1bb5de42 P 059258caec4e43a9858bde7b62fb3ad2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 22:56:10.298015 27406 tablet.cc:2369] T 3f18f698f43f42cca80a60bf1bb5de42 P 4e6df4a857944cee964d912a0760c0da: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 22:56:10.302613 27596 tablet.cc:2369] T 3f18f698f43f42cca80a60bf1bb5de42 P 0c2426a5b37b4aaebafeeb99445d886d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 22:56:10.430853 27635 raft_consensus.cc:482] T 3f18f698f43f42cca80a60bf1bb5de42 P 0c2426a5b37b4aaebafeeb99445d886d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 22:56:10.431202 27635 raft_consensus.cc:504] T 3f18f698f43f42cca80a60bf1bb5de42 P 0c2426a5b37b4aaebafeeb99445d886d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059258caec4e43a9858bde7b62fb3ad2" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 34021 } } peers { permanent_uuid: "4e6df4a857944cee964d912a0760c0da" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 35891 } } peers { permanent_uuid: "0c2426a5b37b4aaebafeeb99445d886d" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 39579 } }
I20260807 22:56:10.433524 27635 leader_election.cc:288] T 3f18f698f43f42cca80a60bf1bb5de42 P 0c2426a5b37b4aaebafeeb99445d886d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 059258caec4e43a9858bde7b62fb3ad2 (127.25.94.1:34021), 4e6df4a857944cee964d912a0760c0da (127.25.94.2:35891)
I20260807 22:56:10.440491 27597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:10.433931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.2:35891 (local address 127.25.94.3:32879)
0807 22:56:10.434361 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0807 22:56:10.434389 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0807 22:56:10.434563 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:56:10.435029 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:56:10.435037 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:56:10.435071 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:56:10.435261 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:10.435275 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:10.436581 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:10.436587 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:10.438127 (+ 1540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:10.438139 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:10.439637 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:10.439646 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:10.439882 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:10.439953 (+ 71us) client_negotiation.cc:769] Sending connection context
0807 22:56:10.440038 (+ 85us) client_negotiation.cc:240] Negotiation successful
0807 22:56:10.440137 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"mutex_wait_us":23}
I20260807 22:56:10.440515 27621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:10.434157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.3:32879 (local address 127.25.94.2:35891)
0807 22:56:10.434390 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:10.434399 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:10.434497 (+ 98us) server_negotiation.cc:407] Connection header received
0807 22:56:10.434739 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:10.434744 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:10.434835 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:10.434958 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:56:10.435461 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:10.436440 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:10.438291 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:10.439423 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:10.439582 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:10.439705 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:10.440097 (+ 392us) server_negotiation.cc:299] Negotiation successful
0807 22:56:10.440244 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260807 22:56:10.440683 27616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:10.434030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.3:35189 (local address 127.25.94.1:34021)
0807 22:56:10.434280 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:10.434288 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:10.434962 (+ 674us) server_negotiation.cc:407] Connection header received
0807 22:56:10.435137 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:10.435142 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:10.435225 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:10.435318 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:56:10.435979 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:10.436835 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:10.438742 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:10.439595 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:10.439734 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:10.439853 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:10.440292 (+ 439us) server_negotiation.cc:299] Negotiation successful
0807 22:56:10.440420 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260807 22:56:10.440842 27639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:10.433937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.1:34021 (local address 127.25.94.3:35189)
0807 22:56:10.434835 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0807 22:56:10.434853 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 22:56:10.434991 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:56:10.435381 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:56:10.435388 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:56:10.435419 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:56:10.435820 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:10.435833 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:10.436986 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:10.436990 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:10.438576 (+ 1586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:10.438592 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:10.439840 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:10.439846 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:10.440109 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:10.440141 (+ 32us) client_negotiation.cc:769] Sending connection context
0807 22:56:10.440232 (+ 91us) client_negotiation.cc:240] Negotiation successful
0807 22:56:10.440390 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":789,"mutex_wait_us":93,"thread_start_us":480,"threads_started":1}
I20260807 22:56:10.442019 27137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f18f698f43f42cca80a60bf1bb5de42" candidate_uuid: "0c2426a5b37b4aaebafeeb99445d886d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059258caec4e43a9858bde7b62fb3ad2" is_pre_election: true
I20260807 22:56:10.442795 27137 raft_consensus.cc:2454] T 3f18f698f43f42cca80a60bf1bb5de42 P 059258caec4e43a9858bde7b62fb3ad2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c2426a5b37b4aaebafeeb99445d886d in term 0.
I20260807 22:56:10.443501 27341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f18f698f43f42cca80a60bf1bb5de42" candidate_uuid: "0c2426a5b37b4aaebafeeb99445d886d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e6df4a857944cee964d912a0760c0da" is_pre_election: true
I20260807 22:56:10.443797 27442 leader_election.cc:302] T 3f18f698f43f42cca80a60bf1bb5de42 P 0c2426a5b37b4aaebafeeb99445d886d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059258caec4e43a9858bde7b62fb3ad2, 0c2426a5b37b4aaebafeeb99445d886d; no voters:
I20260807 22:56:10.444173 27341 raft_consensus.cc:2454] T 3f18f698f43f42cca80a60bf1bb5de42 P 4e6df4a857944cee964d912a0760c0da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c2426a5b37b4aaebafeeb99445d886d in term 0.
I20260807 22:56:10.444271 27635 raft_consensus.cc:2781] T 3f18f698f43f42cca80a60bf1bb5de42 P 0c2426a5b37b4aaebafeeb99445d886d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 22:56:10.444541 27635 raft_consensus.cc:482] T 3f18f698f43f42cca80a60bf1bb5de42 P 0c2426a5b37b4aaebafeeb99445d886d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 22:56:10.444664 27635 raft_consensus.cc:3037] T 3f18f698f43f42cca80a60bf1bb5de42 P 0c2426a5b37b4aaebafeeb99445d886d [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:56:10.447173 27635 raft_consensus.cc:504] T 3f18f698f43f42cca80a60bf1bb5de42 P 0c2426a5b37b4aaebafeeb99445d886d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059258caec4e43a9858bde7b62fb3ad2" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 34021 } } peers { permanent_uuid: "4e6df4a857944cee964d912a0760c0da" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 35891 } } peers { permanent_uuid: "0c2426a5b37b4aaebafeeb99445d886d" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 39579 } }
I20260807 22:56:10.447933 27635 leader_election.cc:288] T 3f18f698f43f42cca80a60bf1bb5de42 P 0c2426a5b37b4aaebafeeb99445d886d [CANDIDATE]: Term 1 election: Requested vote from peers 059258caec4e43a9858bde7b62fb3ad2 (127.25.94.1:34021), 4e6df4a857944cee964d912a0760c0da (127.25.94.2:35891)
I20260807 22:56:10.448618 27137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f18f698f43f42cca80a60bf1bb5de42" candidate_uuid: "0c2426a5b37b4aaebafeeb99445d886d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059258caec4e43a9858bde7b62fb3ad2"
I20260807 22:56:10.448618 27341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f18f698f43f42cca80a60bf1bb5de42" candidate_uuid: "0c2426a5b37b4aaebafeeb99445d886d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e6df4a857944cee964d912a0760c0da"
I20260807 22:56:10.448871 27137 raft_consensus.cc:3037] T 3f18f698f43f42cca80a60bf1bb5de42 P 059258caec4e43a9858bde7b62fb3ad2 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:56:10.448871 27341 raft_consensus.cc:3037] T 3f18f698f43f42cca80a60bf1bb5de42 P 4e6df4a857944cee964d912a0760c0da [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:56:10.452497 27341 raft_consensus.cc:2454] T 3f18f698f43f42cca80a60bf1bb5de42 P 4e6df4a857944cee964d912a0760c0da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c2426a5b37b4aaebafeeb99445d886d in term 1.
I20260807 22:56:10.452498 27137 raft_consensus.cc:2454] T 3f18f698f43f42cca80a60bf1bb5de42 P 059258caec4e43a9858bde7b62fb3ad2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c2426a5b37b4aaebafeeb99445d886d in term 1.
I20260807 22:56:10.453398 27442 leader_election.cc:302] T 3f18f698f43f42cca80a60bf1bb5de42 P 0c2426a5b37b4aaebafeeb99445d886d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 059258caec4e43a9858bde7b62fb3ad2, 0c2426a5b37b4aaebafeeb99445d886d, 4e6df4a857944cee964d912a0760c0da; no voters:
I20260807 22:56:10.453974 27635 raft_consensus.cc:2781] T 3f18f698f43f42cca80a60bf1bb5de42 P 0c2426a5b37b4aaebafeeb99445d886d [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:56:10.454819 27635 raft_consensus.cc:686] T 3f18f698f43f42cca80a60bf1bb5de42 P 0c2426a5b37b4aaebafeeb99445d886d [term 1 LEADER]: Becoming Leader. State: Replica: 0c2426a5b37b4aaebafeeb99445d886d, State: Running, Role: LEADER
I20260807 22:56:10.455564 27635 consensus_queue.cc:237] T 3f18f698f43f42cca80a60bf1bb5de42 P 0c2426a5b37b4aaebafeeb99445d886d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059258caec4e43a9858bde7b62fb3ad2" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 34021 } } peers { permanent_uuid: "4e6df4a857944cee964d912a0760c0da" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 35891 } } peers { permanent_uuid: "0c2426a5b37b4aaebafeeb99445d886d" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 39579 } }
I20260807 22:56:10.461742 26921 catalog_manager.cc:5495] T 3f18f698f43f42cca80a60bf1bb5de42 P 0c2426a5b37b4aaebafeeb99445d886d reported cstate change: term changed from 0 to 1, leader changed from <none> to 0c2426a5b37b4aaebafeeb99445d886d (127.25.94.3). New cstate: current_term: 1 leader_uuid: "0c2426a5b37b4aaebafeeb99445d886d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059258caec4e43a9858bde7b62fb3ad2" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 34021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e6df4a857944cee964d912a0760c0da" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 35891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0c2426a5b37b4aaebafeeb99445d886d" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 39579 } health_report { overall_health: HEALTHY } } }
Aug 07 22:56:10 dist-test-slave-z8x9 krb5kdc[26849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143367, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.1@KRBTEST.COM
I20260807 22:56:10.491145 27616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:10.471805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38472 (local address 127.25.94.1:34021)
0807 22:56:10.471997 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:10.472002 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:10.472546 (+ 544us) server_negotiation.cc:407] Connection header received
0807 22:56:10.472753 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:10.472758 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:10.472855 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:10.472975 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:56:10.475676 (+ 2701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:10.476778 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:10.478153 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:10.478603 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:10.478750 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:10.483432 (+ 4682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:56:10.483450 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:56:10.483468 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:56:10.483510 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:56:10.487383 (+ 3873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:10.488365 (+ 982us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:10.488373 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:10.488376 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:10.488524 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:10.489073 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:10.489077 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:10.489080 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:10.489727 (+ 647us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:56:10.490129 (+ 402us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:10.490469 (+ 340us) server_negotiation.cc:299] Negotiation successful
0807 22:56:10.490731 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
Aug 07 22:56:10 dist-test-slave-z8x9 krb5kdc[26849](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143370, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 07 22:56:10 dist-test-slave-z8x9 krb5kdc[26849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143370, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.25.94.1@KRBTEST.COM
I20260807 22:56:10.544773 27616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:10.528974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38484 (local address 127.25.94.1:34021)
0807 22:56:10.529231 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:10.529238 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:10.529468 (+ 230us) server_negotiation.cc:407] Connection header received
0807 22:56:10.529605 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:10.529609 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:10.529674 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:10.529763 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:56:10.531123 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:10.531922 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:10.533371 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:10.533780 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:10.533881 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:10.538557 (+ 4676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:56:10.538563 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:56:10.538567 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:56:10.538607 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:56:10.541587 (+ 2980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:10.542384 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:10.542408 (+ 24us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:10.542411 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:10.542571 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:10.543044 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:10.543048 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:10.543050 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:10.543528 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:56:10.543924 (+ 396us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:10.544077 (+ 153us) server_negotiation.cc:299] Negotiation successful
0807 22:56:10.544340 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
Aug 07 22:56:10 dist-test-slave-z8x9 krb5kdc[26849](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143370, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 07 22:56:10 dist-test-slave-z8x9 krb5kdc[26849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143370, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.1@KRBTEST.COM
I20260807 22:56:10.600248 27616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:10.583294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38496 (local address 127.25.94.1:34021)
0807 22:56:10.583574 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:10.583581 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:10.583784 (+ 203us) server_negotiation.cc:407] Connection header received
0807 22:56:10.583929 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:10.583933 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:10.584002 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:10.584091 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:56:10.585621 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:10.586324 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:10.587887 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:10.588358 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:10.588456 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:10.593358 (+ 4902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:56:10.593364 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:56:10.593367 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:56:10.593411 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:56:10.596990 (+ 3579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:10.597891 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:10.597896 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:10.597898 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:10.598026 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:10.598470 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:10.598473 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:10.598476 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:10.598825 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:56:10.599088 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:10.599413 (+ 325us) server_negotiation.cc:299] Negotiation successful
0807 22:56:10.599642 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126}
W20260807 22:56:10.602479 27082 tablet_service.cc:2908] Rejecting scan request for tablet 3f18f698f43f42cca80a60bf1bb5de42: Uninitialized: clean time has not yet been initialized
I20260807 22:56:10.606865 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 27011
I20260807 22:56:10.620935 27616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:10.608086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38502 (local address 127.25.94.1:34021)
0807 22:56:10.608241 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:10.608247 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:10.608778 (+ 531us) server_negotiation.cc:407] Connection header received
0807 22:56:10.608925 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:10.608929 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:10.609001 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:10.609085 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:56:10.610585 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:10.611271 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:10.612572 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:10.612976 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:10.613083 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:10.615476 (+ 2393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:56:10.615481 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:56:10.615484 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:56:10.615514 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:56:10.618113 (+ 2599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:10.618863 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:10.618868 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:10.618870 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:10.619080 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:10.619524 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:10.619527 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:10.619529 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:10.619926 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:56:10.620355 (+ 429us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:10.620482 (+ 127us) server_negotiation.cc:299] Negotiation successful
0807 22:56:10.620676 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260807 22:56:10.621706 27191 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 22:56:10.717374 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 27011
I20260807 22:56:10.741977 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 27208
Aug 07 22:56:10 dist-test-slave-z8x9 krb5kdc[26849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143370, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.2@KRBTEST.COM
I20260807 22:56:10.757332 27621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:10.742823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44080 (local address 127.25.94.2:35891)
0807 22:56:10.743012 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:10.743017 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:10.743037 (+ 20us) server_negotiation.cc:407] Connection header received
0807 22:56:10.743159 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:10.743163 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:10.743248 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:10.743333 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:56:10.744472 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:10.745342 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:10.746728 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:10.747143 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:10.747243 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:10.751516 (+ 4273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:56:10.751530 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:56:10.751548 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:56:10.751585 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:56:10.754489 (+ 2904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:10.755226 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:10.755234 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:10.755236 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:10.755379 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:10.755826 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:10.755829 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:10.755831 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:10.756197 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:56:10.756424 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:10.756707 (+ 283us) server_negotiation.cc:299] Negotiation successful
0807 22:56:10.756882 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260807 22:56:10.758083 27399 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 22:56:10.845706 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 27208
I20260807 22:56:10.868832 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 27412
Aug 07 22:56:10 dist-test-slave-z8x9 krb5kdc[26849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143370, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.3@KRBTEST.COM
I20260807 22:56:10.883673 27664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:10.869821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45054 (local address 127.25.94.3:39579)
0807 22:56:10.870273 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:10.870284 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:10.870301 (+ 17us) server_negotiation.cc:407] Connection header received
0807 22:56:10.870356 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:10.870360 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:10.870418 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:10.870515 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:56:10.871734 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:10.872436 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:10.873848 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:10.874241 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:10.874359 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:10.878531 (+ 4172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:56:10.878546 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:56:10.878562 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:56:10.878593 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:56:10.881077 (+ 2484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:10.881775 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:10.881783 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:10.881785 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:10.881969 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:10.882427 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:10.882431 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:10.882433 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:10.882738 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:56:10.882976 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:10.883217 (+ 241us) server_negotiation.cc:299] Negotiation successful
0807 22:56:10.883380 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":245,"threads_started":1}
I20260807 22:56:10.884462 27586 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 22:56:10.974772 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 27412
I20260807 22:56:10.996474 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 26867
I20260807 22:56:10.997900 26967 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 22:56:11.108218 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 26867
2026-08-07T22:56:11Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4739 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786143361779964-25976-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 22:56:11 dist-test-slave-z8x9 krb5kdc[27673](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 22:56:11 dist-test-slave-z8x9 krb5kdc[27673](info): set up 2 sockets
Aug 07 22:56:11 dist-test-slave-z8x9 krb5kdc[27673](info): commencing operation
krb5kdc: starting...
W20260807 22:56:12.453425 25976 mini_kdc.cc:121] Time spent starting KDC: real 1.288s user 0.001s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786143361779964-25976-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786143361779964-25976-0/krb5kdc/test-user.keytab.
Aug 07 22:56:12 dist-test-slave-z8x9 krb5kdc[27673](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143372, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T22:56:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T22:56:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.94.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.62.keytab.
Entry for principal kudu/127.25.94.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.62.keytab.
I20260807 22:56:12.665627 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786143361779964-25976-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786143361779964-25976-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786143361779964-25976-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.62:43863
--webserver_interface=127.25.94.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.62.keytab
--principal=kudu/127.25.94.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.20:36365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.94.62:43863
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786143361779964-25976-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786143361779964-25976-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786143361779964-25976-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:56:12.995055 27691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:56:12.995378 27691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:56:12.995474 27691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:56:13.002640 27691 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 22:56:13.002754 27691 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260807 22:56:13.002797 27691 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:56:13.002857 27691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:56:13.002908 27691 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 22:56:13.002971 27691 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 22:56:13.012913 27691 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:36365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786143361779964-25976-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786143361779964-25976-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.94.62:43863
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.62.keytab
--principal=kudu/127.25.94.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.62:43863
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.94.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786143361779964-25976-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 22:56:13.014319 27691 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:56:13.015681 27691 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:56:13.028195 27697 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:56:13.028110 27700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:56:13.028149 27702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:56:13.028988 27691 server_base.cc:1034] running on GCE node
I20260807 22:56:13.029935 27691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:56:13.031512 27691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:56:13.032835 27691 hybrid_clock.cc:648] HybridClock initialized: now 1786143373032754 us; error 68 us; skew 500 ppm
I20260807 22:56:13.033281 27691 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 22:56:13 dist-test-slave-z8x9 krb5kdc[27673](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143373, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:56:13.036753 27691 init.cc:377] Logged in from keytab as kudu/127.25.94.62@KRBTEST.COM (short username kudu)
I20260807 22:56:13.038549 27691 webserver.cc:460] Webserver started at http://127.25.94.62:39411/ using document root <none> and password file <none>
I20260807 22:56:13.039315 27691 fs_manager.cc:359] Metadata directory not provided
I20260807 22:56:13.039456 27691 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:56:13.039813 27691 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:56:13.042977 27691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786143361779964-25976-0/minicluster-data/master-0/data/instance:
uuid: "00842aecda594fed8f1d657ee997b2c3"
format_stamp: "Formatted at 2026-08-07 22:56:13 on dist-test-slave-z8x9"
server_key: "59e224574f39c2aacce14493ea9e18ef"
server_key_iv: "d6eb3682e3c223be3cc16bd765e426e4"
server_key_version: "encryptionkey@0"
I20260807 22:56:13.043782 27691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786143361779964-25976-0/minicluster-data/master-0/wal/instance:
uuid: "00842aecda594fed8f1d657ee997b2c3"
format_stamp: "Formatted at 2026-08-07 22:56:13 on dist-test-slave-z8x9"
server_key: "59e224574f39c2aacce14493ea9e18ef"
server_key_iv: "d6eb3682e3c223be3cc16bd765e426e4"
server_key_version: "encryptionkey@0"
I20260807 22:56:13.049580 27691 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260807 22:56:13.053287 27714 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:56:13.054682 27691 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 22:56:13.054895 27691 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786143361779964-25976-0/minicluster-data/master-0/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786143361779964-25976-0/minicluster-data/master-0/wal
uuid: "00842aecda594fed8f1d657ee997b2c3"
format_stamp: "Formatted at 2026-08-07 22:56:13 on dist-test-slave-z8x9"
server_key: "59e224574f39c2aacce14493ea9e18ef"
server_key_iv: "d6eb3682e3c223be3cc16bd765e426e4"
server_key_version: "encryptionkey@0"
I20260807 22:56:13.055135 27691 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786143361779964-25976-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786143361779964-25976-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786143361779964-25976-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:56:13.071404 27691 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:56:13.075567 27691 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:56:13.075872 27691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:56:13.099006 27691 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.62:43863
I20260807 22:56:13.099027 27804 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.62:43863 every 8 connection(s)
I20260807 22:56:13.100477 27691 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
I20260807 22:56:13.104420 27805 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:56:13.107380 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 27691
I20260807 22:56:13.107635 25976 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786143361779964-25976-0/minicluster-data/master-0/wal/instance
I20260807 22:56:13.108459 25976 external_mini_cluster.cc:1428] Setting key 73c80e7d6513e880e6cb6eb9c0b432c5
I20260807 22:56:13.118060 27805 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00842aecda594fed8f1d657ee997b2c3: Bootstrap starting.
I20260807 22:56:13.122233 27805 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 00842aecda594fed8f1d657ee997b2c3: Neither blocks nor log segments found. Creating new log.
I20260807 22:56:13.123703 27805 log.cc:824] T 00000000000000000000000000000000 P 00842aecda594fed8f1d657ee997b2c3: Log is configured to *not* fsync() on all Append() calls
Aug 07 22:56:13 dist-test-slave-z8x9 krb5kdc[27673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143372, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.62@KRBTEST.COM
I20260807 22:56:13.127220 27805 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00842aecda594fed8f1d657ee997b2c3: No bootstrap required, opened a new log
I20260807 22:56:13.134025 27810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:13.110763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59220 (local address 127.25.94.62:43863)
0807 22:56:13.111584 (+ 821us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:13.111597 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:13.111639 (+ 42us) server_negotiation.cc:407] Connection header received
0807 22:56:13.113513 (+ 1874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:13.113536 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:13.113963 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:13.114308 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:56:13.116323 (+ 2015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:13.117771 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:13.119590 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:13.120328 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:13.120470 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:13.125971 (+ 5501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:56:13.125980 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:56:13.125996 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:56:13.126032 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:56:13.129359 (+ 3327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:13.130254 (+ 895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:13.130261 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:13.130267 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:13.130462 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:13.130874 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:13.130878 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:13.130879 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:13.131454 (+ 575us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:56:13.131689 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:13.132264 (+ 575us) server_negotiation.cc:299] Negotiation successful
0807 22:56:13.132817 (+ 553us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":611,"thread_start_us":389,"threads_started":1}
I20260807 22:56:13.137135 27805 raft_consensus.cc:348] T 00000000000000000000000000000000 P 00842aecda594fed8f1d657ee997b2c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00842aecda594fed8f1d657ee997b2c3" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 43863 } }
I20260807 22:56:13.137648 27805 raft_consensus.cc:374] T 00000000000000000000000000000000 P 00842aecda594fed8f1d657ee997b2c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:56:13.137789 27805 raft_consensus.cc:729] T 00000000000000000000000000000000 P 00842aecda594fed8f1d657ee997b2c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00842aecda594fed8f1d657ee997b2c3, State: Initialized, Role: FOLLOWER
I20260807 22:56:13.138420 27805 consensus_queue.cc:260] T 00000000000000000000000000000000 P 00842aecda594fed8f1d657ee997b2c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00842aecda594fed8f1d657ee997b2c3" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 43863 } }
I20260807 22:56:13.138679 27805 raft_consensus.cc:388] T 00000000000000000000000000000000 P 00842aecda594fed8f1d657ee997b2c3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 22:56:13.138852 27805 raft_consensus.cc:482] T 00000000000000000000000000000000 P 00842aecda594fed8f1d657ee997b2c3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 22:56:13.139040 27805 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 00842aecda594fed8f1d657ee997b2c3 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:56:13.141731 27805 raft_consensus.cc:504] T 00000000000000000000000000000000 P 00842aecda594fed8f1d657ee997b2c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00842aecda594fed8f1d657ee997b2c3" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 43863 } }
I20260807 22:56:13.142314 27805 leader_election.cc:302] T 00000000000000000000000000000000 P 00842aecda594fed8f1d657ee997b2c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 00842aecda594fed8f1d657ee997b2c3; no voters:
I20260807 22:56:13.142982 27805 leader_election.cc:288] T 00000000000000000000000000000000 P 00842aecda594fed8f1d657ee997b2c3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 22:56:13.143294 27812 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 00842aecda594fed8f1d657ee997b2c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:56:13.144482 27812 raft_consensus.cc:686] T 00000000000000000000000000000000 P 00842aecda594fed8f1d657ee997b2c3 [term 1 LEADER]: Becoming Leader. State: Replica: 00842aecda594fed8f1d657ee997b2c3, State: Running, Role: LEADER
I20260807 22:56:13.145161 27812 consensus_queue.cc:237] T 00000000000000000000000000000000 P 00842aecda594fed8f1d657ee997b2c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00842aecda594fed8f1d657ee997b2c3" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 43863 } }
I20260807 22:56:13.146114 27805 sys_catalog.cc:564] T 00000000000000000000000000000000 P 00842aecda594fed8f1d657ee997b2c3 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 22:56:13.148792 27813 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00842aecda594fed8f1d657ee997b2c3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "00842aecda594fed8f1d657ee997b2c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00842aecda594fed8f1d657ee997b2c3" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 43863 } } }
I20260807 22:56:13.148778 27815 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00842aecda594fed8f1d657ee997b2c3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 00842aecda594fed8f1d657ee997b2c3. Latest consensus state: current_term: 1 leader_uuid: "00842aecda594fed8f1d657ee997b2c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00842aecda594fed8f1d657ee997b2c3" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 43863 } } }
I20260807 22:56:13.149297 27813 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00842aecda594fed8f1d657ee997b2c3 [sys.catalog]: This master's current role is: LEADER
I20260807 22:56:13.149312 27815 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00842aecda594fed8f1d657ee997b2c3 [sys.catalog]: This master's current role is: LEADER
I20260807 22:56:13.151969 27822 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 22:56:13.159193 27822 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 22:56:13.169317 27822 catalog_manager.cc:1347] Generated new cluster ID: 7dfa05e6f6404799bf9181e47ec91c3d
I20260807 22:56:13.169476 27822 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 22:56:13.177796 27822 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 22:56:13.179831 27822 catalog_manager.cc:1504] Loading token signing keys...
I20260807 22:56:13.189333 27822 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 00842aecda594fed8f1d657ee997b2c3: Generated new TSK 0
I20260807 22:56:13.190527 27822 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.94.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.1.keytab.
Entry for principal kudu/127.25.94.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.1.keytab.
I20260807 22:56:13.258015 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786143361779964-25976-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786143361779964-25976-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786143361779964-25976-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786143361779964-25976-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.1:0
--local_ip_for_outbound_sockets=127.25.94.1
--webserver_interface=127.25.94.1
--webserver_port=0
--tserver_master_addrs=127.25.94.62:43863
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.1.keytab
--principal=kudu/127.25.94.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.20:36365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786143361779964-25976-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786143361779964-25976-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786143361779964-25976-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:56:13.605847 27840 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:56:13.606139 27840 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:56:13.606308 27840 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:56:13.613623 27840 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:56:13.613735 27840 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:56:13.613905 27840 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260807 22:56:13.614032 27840 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.1
I20260807 22:56:13.624183 27840 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:36365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786143361779964-25976-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786143361779964-25976-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.1.keytab
--principal=kudu/127.25.94.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786143361779964-25976-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.94.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.25.94.62:43863
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.1
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786143361779964-25976-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 22:56:13.625577 27840 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:56:13.626889 27840 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:56:13.640585 27847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:56:13.640534 27853 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:56:13.640614 27848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:56:13.641451 27840 server_base.cc:1034] running on GCE node
I20260807 22:56:13.642107 27840 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:56:13.643033 27840 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:56:13.644264 27840 hybrid_clock.cc:648] HybridClock initialized: now 1786143373644185 us; error 48 us; skew 500 ppm
I20260807 22:56:13.644609 27840 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 22:56:13 dist-test-slave-z8x9 krb5kdc[27673](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143373, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:56:13.652345 27840 init.cc:377] Logged in from keytab as kudu/127.25.94.1@KRBTEST.COM (short username kudu)
I20260807 22:56:13.654089 27840 webserver.cc:460] Webserver started at http://127.25.94.1:39905/ using document root <none> and password file <none>
I20260807 22:56:13.654829 27840 fs_manager.cc:359] Metadata directory not provided
I20260807 22:56:13.654922 27840 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:56:13.655195 27840 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:56:13.658699 27840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786143361779964-25976-0/minicluster-data/ts-0/data/instance:
uuid: "54b985a98334406b81decd685506beac"
format_stamp: "Formatted at 2026-08-07 22:56:13 on dist-test-slave-z8x9"
server_key: "9af8d02d98a3af5bd569bb75ef0d0beb"
server_key_iv: "dfeb38a9d94943363d003566cb1c003b"
server_key_version: "encryptionkey@0"
I20260807 22:56:13.659722 27840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786143361779964-25976-0/minicluster-data/ts-0/wal/instance:
uuid: "54b985a98334406b81decd685506beac"
format_stamp: "Formatted at 2026-08-07 22:56:13 on dist-test-slave-z8x9"
server_key: "9af8d02d98a3af5bd569bb75ef0d0beb"
server_key_iv: "dfeb38a9d94943363d003566cb1c003b"
server_key_version: "encryptionkey@0"
I20260807 22:56:13.665551 27840 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260807 22:56:13.669034 27865 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:56:13.670507 27840 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 22:56:13.670661 27840 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786143361779964-25976-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786143361779964-25976-0/minicluster-data/ts-0/wal
uuid: "54b985a98334406b81decd685506beac"
format_stamp: "Formatted at 2026-08-07 22:56:13 on dist-test-slave-z8x9"
server_key: "9af8d02d98a3af5bd569bb75ef0d0beb"
server_key_iv: "dfeb38a9d94943363d003566cb1c003b"
server_key_version: "encryptionkey@0"
I20260807 22:56:13.670858 27840 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786143361779964-25976-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786143361779964-25976-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786143361779964-25976-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:56:13.687263 27840 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:56:13.691569 27840 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:56:13.691797 27840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:56:13.694629 27840 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:56:13.694825 27840 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260807 22:56:13.694996 27840 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:56:13.695082 27840 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 07 22:56:13 dist-test-slave-z8x9 krb5kdc[27673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143373, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.1@KRBTEST.COM for kudu/127.25.94.62@KRBTEST.COM
I20260807 22:56:13.716687 27877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:13.695013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.1:39419 (local address 127.25.94.62:43863)
0807 22:56:13.695565 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:13.695573 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:13.696620 (+ 1047us) server_negotiation.cc:407] Connection header received
0807 22:56:13.698696 (+ 2076us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:13.698702 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:13.698788 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:13.698905 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:56:13.701129 (+ 2224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:13.702253 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:13.704900 (+ 2647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:13.705479 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:13.705617 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:13.710529 (+ 4912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:56:13.710534 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:56:13.710538 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:56:13.710572 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:56:13.713158 (+ 2586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:13.714281 (+ 1123us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:13.714290 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:13.714293 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:13.714425 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:13.714885 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:13.714892 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:13.714895 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:13.715166 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:56:13.715323 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:13.716073 (+ 750us) server_negotiation.cc:299] Negotiation successful
0807 22:56:13.716314 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":318,"threads_started":1}
I20260807 22:56:13.718112 27880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:13.695648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:43863 (local address 127.25.94.1:39419)
0807 22:56:13.696451 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0807 22:56:13.696495 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0807 22:56:13.698432 (+ 1937us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:56:13.699143 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:56:13.699151 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:56:13.700056 (+ 905us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:56:13.700928 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:13.700962 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:13.702442 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:13.702446 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:13.704724 (+ 2278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:13.704743 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:13.705736 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:13.705744 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:13.705956 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:13.706849 (+ 893us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:56:13.706884 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:56:13.710239 (+ 3355us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:56:13.713366 (+ 3127us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:56:13.713376 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:56:13.713394 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:56:13.714146 (+ 752us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:56:13.714547 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:56:13.714552 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:56:13.714555 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:56:13.714769 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:56:13.715316 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:56:13.715324 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:56:13.715693 (+ 369us) client_negotiation.cc:769] Sending connection context
0807 22:56:13.715972 (+ 279us) client_negotiation.cc:240] Negotiation successful
0807 22:56:13.716525 (+ 553us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":339,"threads_started":1}
I20260807 22:56:13.741714 27877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:13.726062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.1:53177 (local address 127.25.94.62:43863)
0807 22:56:13.726530 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:13.726539 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:13.727157 (+ 618us) server_negotiation.cc:407] Connection header received
0807 22:56:13.727368 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:13.727373 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:13.727445 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:13.727552 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:56:13.728527 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:13.729318 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:13.731530 (+ 2212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:13.732135 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:13.732285 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:13.735368 (+ 3083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:56:13.735378 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:56:13.735385 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:56:13.735433 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:56:13.739002 (+ 3569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:13.739680 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:13.739686 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:13.739696 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:13.739819 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:13.740231 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:13.740235 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:13.740237 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:13.740512 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:56:13.740669 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:13.741069 (+ 400us) server_negotiation.cc:299] Negotiation successful
0807 22:56:13.741331 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287}
I20260807 22:56:13.741869 27970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:13.726138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:43863 (local address 127.25.94.1:53177)
0807 22:56:13.727042 (+ 904us) negotiation.cc:107] Waiting for socket to connect
0807 22:56:13.727068 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0807 22:56:13.727236 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:56:13.727577 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:56:13.727582 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:56:13.728090 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:56:13.728385 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:13.728394 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:13.729457 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:13.729463 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:13.731322 (+ 1859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:13.731341 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:13.732338 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:13.732345 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:13.732538 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:13.733695 (+ 1157us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:56:13.733721 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:56:13.735130 (+ 1409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:56:13.739140 (+ 4010us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:56:13.739147 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:56:13.739151 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:56:13.739550 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:56:13.739933 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:56:13.739938 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:56:13.739942 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:56:13.740080 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:56:13.740634 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:56:13.740639 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:56:13.740827 (+ 188us) client_negotiation.cc:769] Sending connection context
0807 22:56:13.740977 (+ 150us) client_negotiation.cc:240] Negotiation successful
0807 22:56:13.741370 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":750,"thread_start_us":578,"threads_started":1}
I20260807 22:56:13.756568 27840 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.1:46797
I20260807 22:56:13.756613 28056 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.1:46797 every 8 connection(s)
I20260807 22:56:13.758025 27840 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786143361779964-25976-0/minicluster-data/ts-0/data/info.pb
I20260807 22:56:13.760499 28057 heartbeater.cc:347] Connected to a master server at 127.25.94.62:43863
I20260807 22:56:13.760767 28057 heartbeater.cc:464] Registering TS with master...
I20260807 22:56:13.761472 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 27840
I20260807 22:56:13.761595 28057 heartbeater.cc:511] Master 127.25.94.62:43863 requested a full tablet report, sending...
I20260807 22:56:13.761754 25976 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786143361779964-25976-0/minicluster-data/ts-0/wal/instance
I20260807 22:56:13.762411 25976 external_mini_cluster.cc:1428] Setting key b0d2fa07b2898571ff43915fc52721c1
I20260807 22:56:13.763958 27739 ts_manager.cc:194] Registered new tserver with Master: 54b985a98334406b81decd685506beac (127.25.94.1:46797)
I20260807 22:56:13.766023 27739 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.94.1@KRBTEST.COM'} at 127.25.94.1:39419
WARNING: no policy specified for kudu/127.25.94.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.2.keytab.
Entry for principal kudu/127.25.94.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.2.keytab.
I20260807 22:56:13.805951 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786143361779964-25976-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786143361779964-25976-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786143361779964-25976-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786143361779964-25976-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.2:0
--local_ip_for_outbound_sockets=127.25.94.2
--webserver_interface=127.25.94.2
--webserver_port=0
--tserver_master_addrs=127.25.94.62:43863
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.2.keytab
--principal=kudu/127.25.94.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.20:36365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786143361779964-25976-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786143361779964-25976-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786143361779964-25976-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:56:14.139678 28063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:56:14.140043 28063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:56:14.140216 28063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:56:14.147873 28063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:56:14.148049 28063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:56:14.148193 28063 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260807 22:56:14.148335 28063 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.2
I20260807 22:56:14.158859 28063 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:36365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786143361779964-25976-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786143361779964-25976-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.2.keytab
--principal=kudu/127.25.94.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786143361779964-25976-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.94.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.25.94.62:43863
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.2
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786143361779964-25976-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 22:56:14.160296 28063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:56:14.161767 28063 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:56:14.176226 28070 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:56:14.176252 28075 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:56:14.176314 28073 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:56:14.177102 28063 server_base.cc:1034] running on GCE node
I20260807 22:56:14.177903 28063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:56:14.178866 28063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:56:14.180167 28063 hybrid_clock.cc:648] HybridClock initialized: now 1786143374180051 us; error 99 us; skew 500 ppm
I20260807 22:56:14.180538 28063 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 22:56:14 dist-test-slave-z8x9 krb5kdc[27673](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143374, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:56:14.184155 28063 init.cc:377] Logged in from keytab as kudu/127.25.94.2@KRBTEST.COM (short username kudu)
I20260807 22:56:14.186110 28063 webserver.cc:460] Webserver started at http://127.25.94.2:33445/ using document root <none> and password file <none>
I20260807 22:56:14.186904 28063 fs_manager.cc:359] Metadata directory not provided
I20260807 22:56:14.187050 28063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:56:14.187431 28063 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:56:14.190737 28063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786143361779964-25976-0/minicluster-data/ts-1/data/instance:
uuid: "001a80a933c846228fcb9e7611f26102"
format_stamp: "Formatted at 2026-08-07 22:56:14 on dist-test-slave-z8x9"
server_key: "b8425d4e7fed379072a0484c5a36713c"
server_key_iv: "19247c202e9849bcaef95444907250cf"
server_key_version: "encryptionkey@0"
I20260807 22:56:14.191635 28063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786143361779964-25976-0/minicluster-data/ts-1/wal/instance:
uuid: "001a80a933c846228fcb9e7611f26102"
format_stamp: "Formatted at 2026-08-07 22:56:14 on dist-test-slave-z8x9"
server_key: "b8425d4e7fed379072a0484c5a36713c"
server_key_iv: "19247c202e9849bcaef95444907250cf"
server_key_version: "encryptionkey@0"
I20260807 22:56:14.197340 28063 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260807 22:56:14.201084 28084 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:56:14.202605 28063 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 22:56:14.202805 28063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786143361779964-25976-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786143361779964-25976-0/minicluster-data/ts-1/wal
uuid: "001a80a933c846228fcb9e7611f26102"
format_stamp: "Formatted at 2026-08-07 22:56:14 on dist-test-slave-z8x9"
server_key: "b8425d4e7fed379072a0484c5a36713c"
server_key_iv: "19247c202e9849bcaef95444907250cf"
server_key_version: "encryptionkey@0"
I20260807 22:56:14.203063 28063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786143361779964-25976-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786143361779964-25976-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786143361779964-25976-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:56:14.221814 28063 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:56:14.227506 28063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:56:14.227762 28063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:56:14.230553 28063 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:56:14.230695 28063 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:56:14.230851 28063 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:56:14.230932 28063 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 07 22:56:14 dist-test-slave-z8x9 krb5kdc[27673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143374, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.2@KRBTEST.COM for kudu/127.25.94.62@KRBTEST.COM
I20260807 22:56:14.259487 27877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:14.231459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.2:59675 (local address 127.25.94.62:43863)
0807 22:56:14.231653 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:14.231658 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:14.233310 (+ 1652us) server_negotiation.cc:407] Connection header received
0807 22:56:14.235874 (+ 2564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:14.235884 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:14.235994 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:14.236076 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:56:14.239433 (+ 3357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:14.240586 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:14.243463 (+ 2877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:14.244147 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:14.244313 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:14.250384 (+ 6071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:56:14.250391 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:56:14.250396 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:56:14.250448 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:56:14.254798 (+ 4350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:14.255865 (+ 1067us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:14.255874 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:14.255880 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:14.256075 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:14.256961 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:14.256967 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:14.256970 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:14.257380 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:56:14.257577 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:14.258831 (+ 1254us) server_negotiation.cc:299] Negotiation successful
0807 22:56:14.259089 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260807 22:56:14.261518 28101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:14.232106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:43863 (local address 127.25.94.2:59675)
0807 22:56:14.233074 (+ 968us) negotiation.cc:107] Waiting for socket to connect
0807 22:56:14.233113 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0807 22:56:14.235437 (+ 2324us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:56:14.236623 (+ 1186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:56:14.236641 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:56:14.237583 (+ 942us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:56:14.239157 (+ 1574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:14.239195 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:14.240799 (+ 1604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:14.240803 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:14.243158 (+ 2355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:14.243210 (+ 52us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:14.244511 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:14.244521 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:14.244930 (+ 409us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:14.245897 (+ 967us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:56:14.245940 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:56:14.250004 (+ 4064us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:56:14.255050 (+ 5046us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:56:14.255068 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:56:14.255092 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:56:14.255697 (+ 605us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:56:14.256229 (+ 532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:56:14.256242 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:56:14.256248 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:56:14.256708 (+ 460us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:56:14.257675 (+ 967us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:56:14.257691 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:56:14.258202 (+ 511us) client_negotiation.cc:769] Sending connection context
0807 22:56:14.258705 (+ 503us) client_negotiation.cc:240] Negotiation successful
0807 22:56:14.259405 (+ 700us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":705,"thread_start_us":437,"threads_started":1}
I20260807 22:56:14.276912 28063 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.2:44545
I20260807 22:56:14.276818 28270 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.2:44545 every 8 connection(s)
I20260807 22:56:14.278368 28063 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786143361779964-25976-0/minicluster-data/ts-1/data/info.pb
I20260807 22:56:14.280887 28271 heartbeater.cc:347] Connected to a master server at 127.25.94.62:43863
I20260807 22:56:14.281155 28271 heartbeater.cc:464] Registering TS with master...
I20260807 22:56:14.282084 28271 heartbeater.cc:511] Master 127.25.94.62:43863 requested a full tablet report, sending...
I20260807 22:56:14.283718 27739 ts_manager.cc:194] Registered new tserver with Master: 001a80a933c846228fcb9e7611f26102 (127.25.94.2:44545)
I20260807 22:56:14.284560 27739 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.94.2@KRBTEST.COM'} at 127.25.94.2:59675
I20260807 22:56:14.285696 27877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:14.269895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.2:55819 (local address 127.25.94.62:43863)
0807 22:56:14.270091 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:14.270096 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:14.271243 (+ 1147us) server_negotiation.cc:407] Connection header received
0807 22:56:14.271490 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:14.271496 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:14.271575 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:14.271689 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:56:14.273297 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:14.274380 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:14.276109 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:14.276535 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:14.276762 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:14.279028 (+ 2266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:56:14.279037 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:56:14.279046 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:56:14.279078 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:56:14.282876 (+ 3798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:14.283557 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:14.283564 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:14.283568 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:14.283742 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:14.284160 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:14.284170 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:14.284174 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:14.284531 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:56:14.284707 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:14.285099 (+ 392us) server_negotiation.cc:299] Negotiation successful
0807 22:56:14.285347 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260807 22:56:14.285903 28248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:14.270052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:43863 (local address 127.25.94.2:55819)
0807 22:56:14.271098 (+ 1046us) negotiation.cc:107] Waiting for socket to connect
0807 22:56:14.271116 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 22:56:14.271333 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:56:14.271708 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:56:14.271712 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:56:14.272428 (+ 716us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:56:14.273072 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:14.273085 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:14.274548 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:14.274556 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:14.275936 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:14.275960 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:14.276874 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:14.276883 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:14.277111 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:14.277894 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:56:14.277912 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:56:14.278819 (+ 907us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:56:14.282998 (+ 4179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:56:14.283005 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:56:14.283009 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:56:14.283427 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:56:14.283898 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:56:14.283903 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:56:14.283907 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:56:14.284049 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:56:14.284660 (+ 611us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:56:14.284665 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:56:14.284846 (+ 181us) client_negotiation.cc:769] Sending connection context
0807 22:56:14.284992 (+ 146us) client_negotiation.cc:240] Negotiation successful
0807 22:56:14.285395 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":901,"thread_start_us":682,"threads_started":1}
I20260807 22:56:14.286146 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 28063
I20260807 22:56:14.286358 25976 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786143361779964-25976-0/minicluster-data/ts-1/wal/instance
I20260807 22:56:14.286890 25976 external_mini_cluster.cc:1428] Setting key 9268776455c71dba588a6266701c5b16
WARNING: no policy specified for kudu/127.25.94.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.3.keytab.
Entry for principal kudu/127.25.94.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.3.keytab.
I20260807 22:56:14.333011 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786143361779964-25976-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786143361779964-25976-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786143361779964-25976-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786143361779964-25976-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.3:0
--local_ip_for_outbound_sockets=127.25.94.3
--webserver_interface=127.25.94.3
--webserver_port=0
--tserver_master_addrs=127.25.94.62:43863
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.3.keytab
--principal=kudu/127.25.94.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.20:36365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786143361779964-25976-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786143361779964-25976-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786143361779964-25976-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:56:14.675755 28277 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:56:14.676034 28277 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:56:14.676136 28277 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:56:14.683394 28277 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:56:14.683501 28277 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:56:14.683583 28277 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260807 22:56:14.683665 28277 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.3
I20260807 22:56:14.693303 28277 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:36365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786143361779964-25976-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786143361779964-25976-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.3.keytab
--principal=kudu/127.25.94.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786143361779964-25976-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.94.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.25.94.62:43863
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.3
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786143361779964-25976-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 22:56:14.694523 28277 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:56:14.695789 28277 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:56:14.708894 28289 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:56:14.708966 28286 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:56:14.709692 28285 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:56:14.710191 28277 server_base.cc:1034] running on GCE node
I20260807 22:56:14.710825 28277 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:56:14.711743 28277 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:56:14.712971 28277 hybrid_clock.cc:648] HybridClock initialized: now 1786143374712897 us; error 50 us; skew 500 ppm
I20260807 22:56:14.713377 28277 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 22:56:14 dist-test-slave-z8x9 krb5kdc[27673](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143374, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:56:14.716626 28277 init.cc:377] Logged in from keytab as kudu/127.25.94.3@KRBTEST.COM (short username kudu)
I20260807 22:56:14.718165 28277 webserver.cc:460] Webserver started at http://127.25.94.3:33103/ using document root <none> and password file <none>
I20260807 22:56:14.718919 28277 fs_manager.cc:359] Metadata directory not provided
I20260807 22:56:14.719055 28277 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:56:14.719398 28277 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:56:14.722587 28277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786143361779964-25976-0/minicluster-data/ts-2/data/instance:
uuid: "8e4e220c8d154c4e9f4c68ad7a39efe7"
format_stamp: "Formatted at 2026-08-07 22:56:14 on dist-test-slave-z8x9"
server_key: "ed6bfd688fb578b873712475b1a85c7d"
server_key_iv: "23c4bb5fc74e27b46fa5a5863273993f"
server_key_version: "encryptionkey@0"
I20260807 22:56:14.723423 28277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786143361779964-25976-0/minicluster-data/ts-2/wal/instance:
uuid: "8e4e220c8d154c4e9f4c68ad7a39efe7"
format_stamp: "Formatted at 2026-08-07 22:56:14 on dist-test-slave-z8x9"
server_key: "ed6bfd688fb578b873712475b1a85c7d"
server_key_iv: "23c4bb5fc74e27b46fa5a5863273993f"
server_key_version: "encryptionkey@0"
I20260807 22:56:14.728621 28277 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260807 22:56:14.732187 28301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:56:14.733587 28277 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 22:56:14.733758 28277 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786143361779964-25976-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786143361779964-25976-0/minicluster-data/ts-2/wal
uuid: "8e4e220c8d154c4e9f4c68ad7a39efe7"
format_stamp: "Formatted at 2026-08-07 22:56:14 on dist-test-slave-z8x9"
server_key: "ed6bfd688fb578b873712475b1a85c7d"
server_key_iv: "23c4bb5fc74e27b46fa5a5863273993f"
server_key_version: "encryptionkey@0"
I20260807 22:56:14.733974 28277 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786143361779964-25976-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786143361779964-25976-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786143361779964-25976-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:56:14.754179 28277 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:56:14.759325 28277 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:56:14.759618 28277 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:56:14.763015 28277 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:56:14.763195 28277 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260807 22:56:14.763413 28277 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:56:14.763536 28277 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:56:14.769484 28057 heartbeater.cc:503] Master 127.25.94.62:43863 was elected leader, sending a full tablet report...
Aug 07 22:56:14 dist-test-slave-z8x9 krb5kdc[27673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143374, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.3@KRBTEST.COM for kudu/127.25.94.62@KRBTEST.COM
I20260807 22:56:14.792909 27877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:14.765487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.3:46979 (local address 127.25.94.62:43863)
0807 22:56:14.765660 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:14.765665 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:14.765839 (+ 174us) server_negotiation.cc:407] Connection header received
0807 22:56:14.770950 (+ 5111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:14.770957 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:14.771056 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:14.771171 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:56:14.773665 (+ 2494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:14.774924 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:14.778426 (+ 3502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:14.778850 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:14.778967 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:14.784592 (+ 5625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:56:14.784600 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:56:14.784606 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:56:14.784649 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:56:14.788883 (+ 4234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:14.789971 (+ 1088us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:14.789976 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:14.789978 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:14.790112 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:14.790576 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:14.790582 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:14.790587 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:14.790885 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:56:14.791045 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:14.792076 (+ 1031us) server_negotiation.cc:299] Negotiation successful
0807 22:56:14.792392 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260807 22:56:14.794450 28312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:14.764712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:43863 (local address 127.25.94.3:46979)
0807 22:56:14.765683 (+ 971us) negotiation.cc:107] Waiting for socket to connect
0807 22:56:14.765732 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0807 22:56:14.768632 (+ 2900us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:56:14.771606 (+ 2974us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:56:14.771617 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:56:14.772448 (+ 831us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:56:14.773466 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:14.773496 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:14.775163 (+ 1667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:14.775169 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:14.778228 (+ 3059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:14.778255 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:14.779135 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:14.779145 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:14.779534 (+ 389us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:14.780435 (+ 901us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:56:14.780483 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:56:14.784249 (+ 3766us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:56:14.789073 (+ 4824us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:56:14.789084 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:56:14.789106 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:56:14.789775 (+ 669us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:56:14.790243 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:56:14.790247 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:56:14.790257 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:56:14.790449 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:56:14.791020 (+ 571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:56:14.791028 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:56:14.791540 (+ 512us) client_negotiation.cc:769] Sending connection context
0807 22:56:14.791885 (+ 345us) client_negotiation.cc:240] Negotiation successful
0807 22:56:14.792587 (+ 702us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":658,"thread_start_us":799,"threads_started":1}
I20260807 22:56:14.823437 28277 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.3:38191
I20260807 22:56:14.823496 28473 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.3:38191 every 8 connection(s)
I20260807 22:56:14.824877 27877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:14.807674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.3:50717 (local address 127.25.94.62:43863)
0807 22:56:14.807877 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:14.807882 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:14.808533 (+ 651us) server_negotiation.cc:407] Connection header received
0807 22:56:14.808770 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:14.808776 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:14.808908 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:14.809023 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:56:14.810887 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:14.811853 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:14.814072 (+ 2219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:14.814537 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:14.814681 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:14.817454 (+ 2773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:56:14.817459 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:56:14.817463 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:56:14.817499 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:56:14.821260 (+ 3761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:14.822372 (+ 1112us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:14.822378 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:14.822381 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:14.822585 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:14.823160 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:14.823168 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:14.823172 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:14.823616 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:56:14.823793 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:14.824210 (+ 417us) server_negotiation.cc:299] Negotiation successful
0807 22:56:14.824477 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260807 22:56:14.824947 28419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:14.807814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:43863 (local address 127.25.94.3:50717)
0807 22:56:14.808418 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0807 22:56:14.808435 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 22:56:14.808618 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:56:14.809103 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:56:14.809111 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:56:14.809999 (+ 888us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:56:14.810597 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:14.810618 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:14.812057 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:14.812061 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:14.813829 (+ 1768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:14.813853 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:14.814749 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:14.814758 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:14.814922 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:14.815824 (+ 902us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:56:14.815854 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:56:14.817123 (+ 1269us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:56:14.821656 (+ 4533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:56:14.821664 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:56:14.821668 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:56:14.822224 (+ 556us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:56:14.822763 (+ 539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:56:14.822771 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:56:14.822776 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:56:14.822985 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:56:14.823739 (+ 754us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:56:14.823744 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:56:14.823941 (+ 197us) client_negotiation.cc:769] Sending connection context
0807 22:56:14.824095 (+ 154us) client_negotiation.cc:240] Negotiation successful
0807 22:56:14.824495 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":351,"threads_started":1}
I20260807 22:56:14.825320 28277 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786143361779964-25976-0/minicluster-data/ts-2/data/info.pb
I20260807 22:56:14.828087 28474 heartbeater.cc:347] Connected to a master server at 127.25.94.62:43863
I20260807 22:56:14.828352 28474 heartbeater.cc:464] Registering TS with master...
I20260807 22:56:14.829067 28474 heartbeater.cc:511] Master 127.25.94.62:43863 requested a full tablet report, sending...
I20260807 22:56:14.830464 27738 ts_manager.cc:194] Registered new tserver with Master: 8e4e220c8d154c4e9f4c68ad7a39efe7 (127.25.94.3:38191)
I20260807 22:56:14.831354 27738 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.94.3@KRBTEST.COM'} at 127.25.94.3:46979
I20260807 22:56:14.834829 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 28277
I20260807 22:56:14.835050 25976 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786143361779964-25976-0/minicluster-data/ts-2/wal/instance
I20260807 22:56:14.835515 25976 external_mini_cluster.cc:1428] Setting key c741d742a59f5292595b0e5f9b827657
I20260807 22:56:14.838934 25976 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 07 22:56:14 dist-test-slave-z8x9 krb5kdc[27673](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143374, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 07 22:56:14 dist-test-slave-z8x9 krb5kdc[27673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143374, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.25.94.62@KRBTEST.COM
I20260807 22:56:14.892457 27877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:14.876970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59224 (local address 127.25.94.62:43863)
0807 22:56:14.877158 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:14.877163 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:14.877354 (+ 191us) server_negotiation.cc:407] Connection header received
0807 22:56:14.877513 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:14.877518 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:14.877585 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:14.877681 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:56:14.879200 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:14.879955 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:14.881297 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:14.881721 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:14.881817 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:14.886331 (+ 4514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:56:14.886337 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:56:14.886340 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:56:14.886379 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:56:14.889125 (+ 2746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:14.889921 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:14.889933 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:14.889937 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:14.890071 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:14.890886 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:14.890891 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:14.890893 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:14.891196 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:56:14.891506 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:14.891807 (+ 301us) server_negotiation.cc:299] Negotiation successful
0807 22:56:14.892074 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260807 22:56:14.896817 27738 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59224:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 22:56:14.903614 27738 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 22:56:14.929461 28502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:14.915792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45538 (local address 127.25.94.2:44545)
0807 22:56:14.916465 (+ 673us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:14.916478 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:14.920589 (+ 4111us) server_negotiation.cc:407] Connection header received
0807 22:56:14.920966 (+ 377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:14.920979 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:14.921339 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:14.921564 (+ 225us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:56:14.922591 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:14.924016 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:14.926739 (+ 2723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:14.927979 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:14.928083 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:14.928271 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:14.928915 (+ 644us) server_negotiation.cc:299] Negotiation successful
0807 22:56:14.929096 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":463,"thread_start_us":301,"threads_started":1}
I20260807 22:56:14.929561 28499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:14.915304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.2:44545 (local address 127.0.0.1:45538)
0807 22:56:14.920436 (+ 5132us) negotiation.cc:107] Waiting for socket to connect
0807 22:56:14.920459 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0807 22:56:14.920725 (+ 266us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:56:14.921917 (+ 1192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:56:14.921925 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:56:14.921996 (+ 71us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:56:14.922391 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:14.922410 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:14.924141 (+ 1731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:14.924150 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:14.926571 (+ 2421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:14.926587 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:14.928431 (+ 1844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:14.928437 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:14.928641 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:14.928694 (+ 53us) client_negotiation.cc:769] Sending connection context
0807 22:56:14.928864 (+ 170us) client_negotiation.cc:240] Negotiation successful
0807 22:56:14.929103 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4908,"mutex_wait_us":149,"spinlock_wait_cycles":1920,"thread_start_us":4697,"threads_started":1}
I20260807 22:56:14.929654 28503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:14.915966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45286 (local address 127.25.94.1:46797)
0807 22:56:14.918843 (+ 2877us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:14.918854 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:14.920835 (+ 1981us) server_negotiation.cc:407] Connection header received
0807 22:56:14.921140 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:14.921150 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:14.921425 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:14.921609 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:56:14.922727 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:14.924196 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:14.926489 (+ 2293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:14.927901 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:14.928043 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:14.928258 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:14.929144 (+ 886us) server_negotiation.cc:299] Negotiation successful
0807 22:56:14.929360 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2671,"thread_start_us":2513,"threads_started":1}
I20260807 22:56:14.930282 28504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:14.916250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.1:46797 (local address 127.0.0.1:45286)
0807 22:56:14.920716 (+ 4466us) negotiation.cc:107] Waiting for socket to connect
0807 22:56:14.920733 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 22:56:14.920934 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:56:14.921917 (+ 983us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:56:14.921925 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:56:14.921967 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:56:14.922544 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:14.922561 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:14.924381 (+ 1820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:14.924386 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:14.926328 (+ 1942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:14.926340 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:14.928590 (+ 2250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:14.928598 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:14.928743 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:14.928776 (+ 33us) client_negotiation.cc:769] Sending connection context
0807 22:56:14.928869 (+ 93us) client_negotiation.cc:240] Negotiation successful
0807 22:56:14.929979 (+ 1110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4347,"mutex_wait_us":909,"spinlock_wait_cycles":1792,"thread_start_us":4205,"threads_started":1}
I20260807 22:56:14.930604 28498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:14.915177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.3:38191 (local address 127.0.0.1:56680)
0807 22:56:14.919393 (+ 4216us) negotiation.cc:107] Waiting for socket to connect
0807 22:56:14.919429 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0807 22:56:14.919594 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:56:14.922665 (+ 3071us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:56:14.922671 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:56:14.922697 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:56:14.923060 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:14.923073 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:14.925510 (+ 2437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:14.925516 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:14.927530 (+ 2014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:14.927546 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:14.929666 (+ 2120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:14.929675 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:14.929844 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:14.929878 (+ 34us) client_negotiation.cc:769] Sending connection context
0807 22:56:14.929967 (+ 89us) client_negotiation.cc:240] Negotiation successful
0807 22:56:14.930111 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3971,"mutex_wait_us":42,"thread_start_us":3838,"threads_started":1}
I20260807 22:56:14.930863 28501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:14.915659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56680 (local address 127.25.94.3:38191)
0807 22:56:14.922096 (+ 6437us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:14.922107 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:14.922141 (+ 34us) server_negotiation.cc:407] Connection header received
0807 22:56:14.922237 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:14.922248 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:14.922475 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:14.922669 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:56:14.923282 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:14.925357 (+ 2075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:14.927717 (+ 2360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:14.929082 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:14.929243 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:14.929490 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:14.930381 (+ 891us) server_negotiation.cc:299] Negotiation successful
0807 22:56:14.930540 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6255,"thread_start_us":6029,"threads_started":1}
I20260807 22:56:14.933712 28163 tablet_service.cc:1467] Processing CreateTablet for tablet 24c39a4277894fac8184f75cd9da6cd4 (DEFAULT_TABLE table=test-table [id=16a4d4571fa94e1abcb460ebe7523d4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:56:14.933957 27941 tablet_service.cc:1467] Processing CreateTablet for tablet 24c39a4277894fac8184f75cd9da6cd4 (DEFAULT_TABLE table=test-table [id=16a4d4571fa94e1abcb460ebe7523d4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:56:14.934957 28163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24c39a4277894fac8184f75cd9da6cd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:56:14.935348 27941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24c39a4277894fac8184f75cd9da6cd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:56:14.935683 28369 tablet_service.cc:1467] Processing CreateTablet for tablet 24c39a4277894fac8184f75cd9da6cd4 (DEFAULT_TABLE table=test-table [id=16a4d4571fa94e1abcb460ebe7523d4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:56:14.937091 28369 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24c39a4277894fac8184f75cd9da6cd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:56:14.946657 28506 tablet_bootstrap.cc:492] T 24c39a4277894fac8184f75cd9da6cd4 P 001a80a933c846228fcb9e7611f26102: Bootstrap starting.
I20260807 22:56:14.949867 28509 tablet_bootstrap.cc:492] T 24c39a4277894fac8184f75cd9da6cd4 P 54b985a98334406b81decd685506beac: Bootstrap starting.
I20260807 22:56:14.949980 28506 tablet_bootstrap.cc:654] T 24c39a4277894fac8184f75cd9da6cd4 P 001a80a933c846228fcb9e7611f26102: Neither blocks nor log segments found. Creating new log.
I20260807 22:56:14.951733 28506 log.cc:824] T 24c39a4277894fac8184f75cd9da6cd4 P 001a80a933c846228fcb9e7611f26102: Log is configured to *not* fsync() on all Append() calls
I20260807 22:56:14.952325 28512 tablet_bootstrap.cc:492] T 24c39a4277894fac8184f75cd9da6cd4 P 8e4e220c8d154c4e9f4c68ad7a39efe7: Bootstrap starting.
I20260807 22:56:14.954178 28509 tablet_bootstrap.cc:654] T 24c39a4277894fac8184f75cd9da6cd4 P 54b985a98334406b81decd685506beac: Neither blocks nor log segments found. Creating new log.
I20260807 22:56:14.956251 28509 log.cc:824] T 24c39a4277894fac8184f75cd9da6cd4 P 54b985a98334406b81decd685506beac: Log is configured to *not* fsync() on all Append() calls
I20260807 22:56:14.956283 28512 tablet_bootstrap.cc:654] T 24c39a4277894fac8184f75cd9da6cd4 P 8e4e220c8d154c4e9f4c68ad7a39efe7: Neither blocks nor log segments found. Creating new log.
I20260807 22:56:14.956398 28506 tablet_bootstrap.cc:492] T 24c39a4277894fac8184f75cd9da6cd4 P 001a80a933c846228fcb9e7611f26102: No bootstrap required, opened a new log
I20260807 22:56:14.956692 28506 ts_tablet_manager.cc:1397] T 24c39a4277894fac8184f75cd9da6cd4 P 001a80a933c846228fcb9e7611f26102: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.008s
I20260807 22:56:14.957756 28512 log.cc:824] T 24c39a4277894fac8184f75cd9da6cd4 P 8e4e220c8d154c4e9f4c68ad7a39efe7: Log is configured to *not* fsync() on all Append() calls
I20260807 22:56:14.959734 28509 tablet_bootstrap.cc:492] T 24c39a4277894fac8184f75cd9da6cd4 P 54b985a98334406b81decd685506beac: No bootstrap required, opened a new log
I20260807 22:56:14.960105 28509 ts_tablet_manager.cc:1397] T 24c39a4277894fac8184f75cd9da6cd4 P 54b985a98334406b81decd685506beac: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.004s
I20260807 22:56:14.960764 28512 tablet_bootstrap.cc:492] T 24c39a4277894fac8184f75cd9da6cd4 P 8e4e220c8d154c4e9f4c68ad7a39efe7: No bootstrap required, opened a new log
I20260807 22:56:14.961023 28512 ts_tablet_manager.cc:1397] T 24c39a4277894fac8184f75cd9da6cd4 P 8e4e220c8d154c4e9f4c68ad7a39efe7: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260807 22:56:14.968470 28506 raft_consensus.cc:348] T 24c39a4277894fac8184f75cd9da6cd4 P 001a80a933c846228fcb9e7611f26102 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e4e220c8d154c4e9f4c68ad7a39efe7" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 38191 } } peers { permanent_uuid: "001a80a933c846228fcb9e7611f26102" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 44545 } } peers { permanent_uuid: "54b985a98334406b81decd685506beac" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 46797 } }
I20260807 22:56:14.969117 28506 raft_consensus.cc:374] T 24c39a4277894fac8184f75cd9da6cd4 P 001a80a933c846228fcb9e7611f26102 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:56:14.969532 28506 raft_consensus.cc:729] T 24c39a4277894fac8184f75cd9da6cd4 P 001a80a933c846228fcb9e7611f26102 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 001a80a933c846228fcb9e7611f26102, State: Initialized, Role: FOLLOWER
I20260807 22:56:14.970243 28512 raft_consensus.cc:348] T 24c39a4277894fac8184f75cd9da6cd4 P 8e4e220c8d154c4e9f4c68ad7a39efe7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e4e220c8d154c4e9f4c68ad7a39efe7" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 38191 } } peers { permanent_uuid: "001a80a933c846228fcb9e7611f26102" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 44545 } } peers { permanent_uuid: "54b985a98334406b81decd685506beac" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 46797 } }
I20260807 22:56:14.970397 28506 consensus_queue.cc:260] T 24c39a4277894fac8184f75cd9da6cd4 P 001a80a933c846228fcb9e7611f26102 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e4e220c8d154c4e9f4c68ad7a39efe7" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 38191 } } peers { permanent_uuid: "001a80a933c846228fcb9e7611f26102" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 44545 } } peers { permanent_uuid: "54b985a98334406b81decd685506beac" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 46797 } }
I20260807 22:56:14.970762 28512 raft_consensus.cc:374] T 24c39a4277894fac8184f75cd9da6cd4 P 8e4e220c8d154c4e9f4c68ad7a39efe7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:56:14.970917 28512 raft_consensus.cc:729] T 24c39a4277894fac8184f75cd9da6cd4 P 8e4e220c8d154c4e9f4c68ad7a39efe7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e4e220c8d154c4e9f4c68ad7a39efe7, State: Initialized, Role: FOLLOWER
I20260807 22:56:14.971635 28512 consensus_queue.cc:260] T 24c39a4277894fac8184f75cd9da6cd4 P 8e4e220c8d154c4e9f4c68ad7a39efe7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e4e220c8d154c4e9f4c68ad7a39efe7" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 38191 } } peers { permanent_uuid: "001a80a933c846228fcb9e7611f26102" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 44545 } } peers { permanent_uuid: "54b985a98334406b81decd685506beac" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 46797 } }
I20260807 22:56:14.971701 28509 raft_consensus.cc:348] T 24c39a4277894fac8184f75cd9da6cd4 P 54b985a98334406b81decd685506beac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e4e220c8d154c4e9f4c68ad7a39efe7" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 38191 } } peers { permanent_uuid: "001a80a933c846228fcb9e7611f26102" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 44545 } } peers { permanent_uuid: "54b985a98334406b81decd685506beac" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 46797 } }
I20260807 22:56:14.972251 28509 raft_consensus.cc:374] T 24c39a4277894fac8184f75cd9da6cd4 P 54b985a98334406b81decd685506beac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:56:14.972420 28509 raft_consensus.cc:729] T 24c39a4277894fac8184f75cd9da6cd4 P 54b985a98334406b81decd685506beac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54b985a98334406b81decd685506beac, State: Initialized, Role: FOLLOWER
I20260807 22:56:14.972713 28271 heartbeater.cc:503] Master 127.25.94.62:43863 was elected leader, sending a full tablet report...
I20260807 22:56:14.973116 28474 heartbeater.cc:503] Master 127.25.94.62:43863 was elected leader, sending a full tablet report...
I20260807 22:56:14.973145 28509 consensus_queue.cc:260] T 24c39a4277894fac8184f75cd9da6cd4 P 54b985a98334406b81decd685506beac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e4e220c8d154c4e9f4c68ad7a39efe7" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 38191 } } peers { permanent_uuid: "001a80a933c846228fcb9e7611f26102" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 44545 } } peers { permanent_uuid: "54b985a98334406b81decd685506beac" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 46797 } }
I20260807 22:56:14.973505 28506 ts_tablet_manager.cc:1428] T 24c39a4277894fac8184f75cd9da6cd4 P 001a80a933c846228fcb9e7611f26102: Time spent starting tablet: real 0.017s user 0.016s sys 0.000s
I20260807 22:56:14.975533 28509 ts_tablet_manager.cc:1428] T 24c39a4277894fac8184f75cd9da6cd4 P 54b985a98334406b81decd685506beac: Time spent starting tablet: real 0.015s user 0.011s sys 0.003s
I20260807 22:56:14.975672 28512 ts_tablet_manager.cc:1428] T 24c39a4277894fac8184f75cd9da6cd4 P 8e4e220c8d154c4e9f4c68ad7a39efe7: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260807 22:56:15.009719 28518 raft_consensus.cc:482] T 24c39a4277894fac8184f75cd9da6cd4 P 001a80a933c846228fcb9e7611f26102 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 22:56:15.010066 28518 raft_consensus.cc:504] T 24c39a4277894fac8184f75cd9da6cd4 P 001a80a933c846228fcb9e7611f26102 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e4e220c8d154c4e9f4c68ad7a39efe7" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 38191 } } peers { permanent_uuid: "001a80a933c846228fcb9e7611f26102" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 44545 } } peers { permanent_uuid: "54b985a98334406b81decd685506beac" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 46797 } }
W20260807 22:56:15.010605 28058 tablet.cc:2369] T 24c39a4277894fac8184f75cd9da6cd4 P 54b985a98334406b81decd685506beac: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 22:56:15.011830 28518 leader_election.cc:288] T 24c39a4277894fac8184f75cd9da6cd4 P 001a80a933c846228fcb9e7611f26102 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8e4e220c8d154c4e9f4c68ad7a39efe7 (127.25.94.3:38191), 54b985a98334406b81decd685506beac (127.25.94.1:46797)
I20260807 22:56:15.021442 28501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:15.012448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.2:51963 (local address 127.25.94.3:38191)
0807 22:56:15.012685 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:15.012692 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:15.013469 (+ 777us) server_negotiation.cc:407] Connection header received
0807 22:56:15.013667 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:15.013673 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:15.013760 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:15.013867 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:56:15.014911 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:15.015965 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:15.018984 (+ 3019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:15.020280 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:15.020414 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:15.020521 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:15.021022 (+ 501us) server_negotiation.cc:299] Negotiation successful
0807 22:56:15.021157 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260807 22:56:15.021443 28503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:15.012457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.2:59679 (local address 127.25.94.1:46797)
0807 22:56:15.012655 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:15.012674 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:15.013469 (+ 795us) server_negotiation.cc:407] Connection header received
0807 22:56:15.013667 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:15.013673 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:15.013760 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:15.013867 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:56:15.014776 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:15.015940 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:15.018984 (+ 3044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:15.020281 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:15.020414 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:15.020521 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:15.020978 (+ 457us) server_negotiation.cc:299] Negotiation successful
0807 22:56:15.021129 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260807 22:56:15.021796 28523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:15.012264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.3:38191 (local address 127.25.94.2:51963)
0807 22:56:15.013352 (+ 1088us) negotiation.cc:107] Waiting for socket to connect
0807 22:56:15.013370 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 22:56:15.013512 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:56:15.013938 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:56:15.013944 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:56:15.014005 (+ 61us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:56:15.014747 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:15.014759 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:15.016099 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:15.016105 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:15.018826 (+ 2721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:15.018836 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:15.020685 (+ 1849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:15.020690 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:15.020834 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:15.020857 (+ 23us) client_negotiation.cc:769] Sending connection context
0807 22:56:15.020974 (+ 117us) client_negotiation.cc:240] Negotiation successful
0807 22:56:15.021194 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":987,"mutex_wait_us":221,"spinlock_wait_cycles":14208,"thread_start_us":630,"threads_started":1}
I20260807 22:56:15.022158 28524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:15.012149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.1:46797 (local address 127.25.94.2:59679)
0807 22:56:15.013319 (+ 1170us) negotiation.cc:107] Waiting for socket to connect
0807 22:56:15.013345 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0807 22:56:15.013520 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:56:15.013938 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:56:15.013944 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:56:15.014005 (+ 61us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:56:15.014610 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:15.014627 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:15.016101 (+ 1474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:15.016105 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:15.018805 (+ 2700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:15.018826 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:15.020650 (+ 1824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:15.020659 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:15.020805 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:15.020834 (+ 29us) client_negotiation.cc:769] Sending connection context
0807 22:56:15.020916 (+ 82us) client_negotiation.cc:240] Negotiation successful
0807 22:56:15.021105 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":979,"mutex_wait_us":262,"spinlock_wait_cycles":85248,"thread_start_us":756,"threads_started":1}
I20260807 22:56:15.023084 28408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24c39a4277894fac8184f75cd9da6cd4" candidate_uuid: "001a80a933c846228fcb9e7611f26102" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e4e220c8d154c4e9f4c68ad7a39efe7" is_pre_election: true
I20260807 22:56:15.023157 27982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24c39a4277894fac8184f75cd9da6cd4" candidate_uuid: "001a80a933c846228fcb9e7611f26102" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54b985a98334406b81decd685506beac" is_pre_election: true
I20260807 22:56:15.023640 27982 raft_consensus.cc:2454] T 24c39a4277894fac8184f75cd9da6cd4 P 54b985a98334406b81decd685506beac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 001a80a933c846228fcb9e7611f26102 in term 0.
I20260807 22:56:15.023617 28408 raft_consensus.cc:2454] T 24c39a4277894fac8184f75cd9da6cd4 P 8e4e220c8d154c4e9f4c68ad7a39efe7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 001a80a933c846228fcb9e7611f26102 in term 0.
I20260807 22:56:15.024353 28088 leader_election.cc:302] T 24c39a4277894fac8184f75cd9da6cd4 P 001a80a933c846228fcb9e7611f26102 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 001a80a933c846228fcb9e7611f26102, 54b985a98334406b81decd685506beac, 8e4e220c8d154c4e9f4c68ad7a39efe7; no voters:
I20260807 22:56:15.024823 28518 raft_consensus.cc:2781] T 24c39a4277894fac8184f75cd9da6cd4 P 001a80a933c846228fcb9e7611f26102 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 22:56:15.024993 28518 raft_consensus.cc:482] T 24c39a4277894fac8184f75cd9da6cd4 P 001a80a933c846228fcb9e7611f26102 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 22:56:15.025118 28518 raft_consensus.cc:3037] T 24c39a4277894fac8184f75cd9da6cd4 P 001a80a933c846228fcb9e7611f26102 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:56:15.027631 28518 raft_consensus.cc:504] T 24c39a4277894fac8184f75cd9da6cd4 P 001a80a933c846228fcb9e7611f26102 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e4e220c8d154c4e9f4c68ad7a39efe7" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 38191 } } peers { permanent_uuid: "001a80a933c846228fcb9e7611f26102" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 44545 } } peers { permanent_uuid: "54b985a98334406b81decd685506beac" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 46797 } }
I20260807 22:56:15.028228 28518 leader_election.cc:288] T 24c39a4277894fac8184f75cd9da6cd4 P 001a80a933c846228fcb9e7611f26102 [CANDIDATE]: Term 1 election: Requested vote from peers 8e4e220c8d154c4e9f4c68ad7a39efe7 (127.25.94.3:38191), 54b985a98334406b81decd685506beac (127.25.94.1:46797)
I20260807 22:56:15.028813 28408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24c39a4277894fac8184f75cd9da6cd4" candidate_uuid: "001a80a933c846228fcb9e7611f26102" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e4e220c8d154c4e9f4c68ad7a39efe7"
I20260807 22:56:15.028774 27982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24c39a4277894fac8184f75cd9da6cd4" candidate_uuid: "001a80a933c846228fcb9e7611f26102" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54b985a98334406b81decd685506beac"
I20260807 22:56:15.029031 27982 raft_consensus.cc:3037] T 24c39a4277894fac8184f75cd9da6cd4 P 54b985a98334406b81decd685506beac [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:56:15.029031 28408 raft_consensus.cc:3037] T 24c39a4277894fac8184f75cd9da6cd4 P 8e4e220c8d154c4e9f4c68ad7a39efe7 [term 0 FOLLOWER]: Advancing to term 1
W20260807 22:56:15.030807 28272 tablet.cc:2369] T 24c39a4277894fac8184f75cd9da6cd4 P 001a80a933c846228fcb9e7611f26102: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 22:56:15.033200 27982 raft_consensus.cc:2454] T 24c39a4277894fac8184f75cd9da6cd4 P 54b985a98334406b81decd685506beac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 001a80a933c846228fcb9e7611f26102 in term 1.
I20260807 22:56:15.033639 28088 leader_election.cc:302] T 24c39a4277894fac8184f75cd9da6cd4 P 001a80a933c846228fcb9e7611f26102 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 001a80a933c846228fcb9e7611f26102, 54b985a98334406b81decd685506beac; no voters:
I20260807 22:56:15.033715 28408 raft_consensus.cc:2454] T 24c39a4277894fac8184f75cd9da6cd4 P 8e4e220c8d154c4e9f4c68ad7a39efe7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 001a80a933c846228fcb9e7611f26102 in term 1.
I20260807 22:56:15.033988 28518 raft_consensus.cc:2781] T 24c39a4277894fac8184f75cd9da6cd4 P 001a80a933c846228fcb9e7611f26102 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:56:15.034571 28518 raft_consensus.cc:686] T 24c39a4277894fac8184f75cd9da6cd4 P 001a80a933c846228fcb9e7611f26102 [term 1 LEADER]: Becoming Leader. State: Replica: 001a80a933c846228fcb9e7611f26102, State: Running, Role: LEADER
I20260807 22:56:15.035110 28518 consensus_queue.cc:237] T 24c39a4277894fac8184f75cd9da6cd4 P 001a80a933c846228fcb9e7611f26102 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e4e220c8d154c4e9f4c68ad7a39efe7" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 38191 } } peers { permanent_uuid: "001a80a933c846228fcb9e7611f26102" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 44545 } } peers { permanent_uuid: "54b985a98334406b81decd685506beac" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 46797 } }
I20260807 22:56:15.039948 27738 catalog_manager.cc:5495] T 24c39a4277894fac8184f75cd9da6cd4 P 001a80a933c846228fcb9e7611f26102 reported cstate change: term changed from 0 to 1, leader changed from <none> to 001a80a933c846228fcb9e7611f26102 (127.25.94.2). New cstate: current_term: 1 leader_uuid: "001a80a933c846228fcb9e7611f26102" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e4e220c8d154c4e9f4c68ad7a39efe7" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 38191 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "001a80a933c846228fcb9e7611f26102" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 44545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54b985a98334406b81decd685506beac" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 46797 } health_report { overall_health: UNKNOWN } } }
I20260807 22:56:15.069653 28502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:15.063923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45542 (local address 127.25.94.2:44545)
0807 22:56:15.064206 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:15.064210 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:15.064237 (+ 27us) server_negotiation.cc:407] Connection header received
0807 22:56:15.064501 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:15.064506 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:15.064601 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:15.064733 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 22:56:15.065122 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:15.065886 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:15.067420 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:15.067838 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:15.067934 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:15.068424 (+ 490us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 22:56:15.069023 (+ 599us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 22:56:15.069130 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:15.069283 (+ 153us) server_negotiation.cc:299] Negotiation successful
0807 22:56:15.069349 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260807 22:56:15.079548 27982 raft_consensus.cc:1261] T 24c39a4277894fac8184f75cd9da6cd4 P 54b985a98334406b81decd685506beac [term 1 FOLLOWER]: Refusing update from remote peer 001a80a933c846228fcb9e7611f26102: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 22:56:15.080850 28525 consensus_queue.cc:1035] T 24c39a4277894fac8184f75cd9da6cd4 P 001a80a933c846228fcb9e7611f26102 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54b985a98334406b81decd685506beac" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 46797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 22:56:15.081450 28408 raft_consensus.cc:1261] T 24c39a4277894fac8184f75cd9da6cd4 P 8e4e220c8d154c4e9f4c68ad7a39efe7 [term 1 FOLLOWER]: Refusing update from remote peer 001a80a933c846228fcb9e7611f26102: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 22:56:15.082376 28525 consensus_queue.cc:1035] T 24c39a4277894fac8184f75cd9da6cd4 P 001a80a933c846228fcb9e7611f26102 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e4e220c8d154c4e9f4c68ad7a39efe7" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 38191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260807 22:56:15.082947 28476 tablet.cc:2369] T 24c39a4277894fac8184f75cd9da6cd4 P 8e4e220c8d154c4e9f4c68ad7a39efe7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 22:56:15.121995 27739 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59224:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 22:56:15.122316 27739 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59224:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 22:56:15.127452 27739 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 00842aecda594fed8f1d657ee997b2c3: Sending DeleteTablet for 3 replicas of tablet 24c39a4277894fac8184f75cd9da6cd4
I20260807 22:56:15.129033 28369 tablet_service.cc:1514] Processing DeleteTablet for tablet 24c39a4277894fac8184f75cd9da6cd4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:56:15 UTC) from {username='kudu', principal='kudu/127.25.94.62@KRBTEST.COM'} at 127.0.0.1:56680
I20260807 22:56:15.129294 27941 tablet_service.cc:1514] Processing DeleteTablet for tablet 24c39a4277894fac8184f75cd9da6cd4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:56:15 UTC) from {username='kudu', principal='kudu/127.25.94.62@KRBTEST.COM'} at 127.0.0.1:45286
I20260807 22:56:15.129509 28163 tablet_service.cc:1514] Processing DeleteTablet for tablet 24c39a4277894fac8184f75cd9da6cd4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:56:15 UTC) from {username='kudu', principal='kudu/127.25.94.62@KRBTEST.COM'} at 127.0.0.1:45538
I20260807 22:56:15.135978 27739 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59224:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 22:56:15.136826 27739 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 22:56:15.142030 28551 tablet_replica.cc:330] T 24c39a4277894fac8184f75cd9da6cd4 P 8e4e220c8d154c4e9f4c68ad7a39efe7: stopping tablet replica
I20260807 22:56:15.142793 28552 tablet_replica.cc:330] T 24c39a4277894fac8184f75cd9da6cd4 P 54b985a98334406b81decd685506beac: stopping tablet replica
I20260807 22:56:15.143509 28552 raft_consensus.cc:2229] T 24c39a4277894fac8184f75cd9da6cd4 P 54b985a98334406b81decd685506beac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 22:56:15.144093 28551 raft_consensus.cc:2229] T 24c39a4277894fac8184f75cd9da6cd4 P 8e4e220c8d154c4e9f4c68ad7a39efe7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 22:56:15.144731 28552 raft_consensus.cc:2258] T 24c39a4277894fac8184f75cd9da6cd4 P 54b985a98334406b81decd685506beac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 22:56:15.145351 28551 raft_consensus.cc:2258] T 24c39a4277894fac8184f75cd9da6cd4 P 8e4e220c8d154c4e9f4c68ad7a39efe7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 22:56:15.148725 28551 ts_tablet_manager.cc:1905] T 24c39a4277894fac8184f75cd9da6cd4 P 8e4e220c8d154c4e9f4c68ad7a39efe7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 22:56:15.150611 28552 ts_tablet_manager.cc:1905] T 24c39a4277894fac8184f75cd9da6cd4 P 54b985a98334406b81decd685506beac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 22:56:15.151096 27941 tablet_service.cc:1467] Processing CreateTablet for tablet ce237cf27e5a4781a47817430cc184ed (DEFAULT_TABLE table=test-table [id=4a7eef5298dc48d59cfec10fa50546e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:56:15.152416 28162 tablet_service.cc:1467] Processing CreateTablet for tablet ce237cf27e5a4781a47817430cc184ed (DEFAULT_TABLE table=test-table [id=4a7eef5298dc48d59cfec10fa50546e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:56:15.153322 28162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce237cf27e5a4781a47817430cc184ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:56:15.156220 27941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce237cf27e5a4781a47817430cc184ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:56:15.160445 28368 tablet_service.cc:1467] Processing CreateTablet for tablet ce237cf27e5a4781a47817430cc184ed (DEFAULT_TABLE table=test-table [id=4a7eef5298dc48d59cfec10fa50546e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:56:15.161396 28368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce237cf27e5a4781a47817430cc184ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:56:15.168820 28512 tablet_bootstrap.cc:492] T ce237cf27e5a4781a47817430cc184ed P 8e4e220c8d154c4e9f4c68ad7a39efe7: Bootstrap starting.
I20260807 22:56:15.170915 28512 tablet_bootstrap.cc:654] T ce237cf27e5a4781a47817430cc184ed P 8e4e220c8d154c4e9f4c68ad7a39efe7: Neither blocks nor log segments found. Creating new log.
I20260807 22:56:15.184316 28509 tablet_bootstrap.cc:492] T ce237cf27e5a4781a47817430cc184ed P 54b985a98334406b81decd685506beac: Bootstrap starting.
I20260807 22:56:15.186992 28509 tablet_bootstrap.cc:654] T ce237cf27e5a4781a47817430cc184ed P 54b985a98334406b81decd685506beac: Neither blocks nor log segments found. Creating new log.
I20260807 22:56:15.190912 28512 tablet_bootstrap.cc:492] T ce237cf27e5a4781a47817430cc184ed P 8e4e220c8d154c4e9f4c68ad7a39efe7: No bootstrap required, opened a new log
I20260807 22:56:15.191197 28512 ts_tablet_manager.cc:1397] T ce237cf27e5a4781a47817430cc184ed P 8e4e220c8d154c4e9f4c68ad7a39efe7: Time spent bootstrapping tablet: real 0.022s user 0.006s sys 0.004s
I20260807 22:56:15.192257 28512 raft_consensus.cc:348] T ce237cf27e5a4781a47817430cc184ed P 8e4e220c8d154c4e9f4c68ad7a39efe7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54b985a98334406b81decd685506beac" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 46797 } } peers { permanent_uuid: "001a80a933c846228fcb9e7611f26102" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 44545 } } peers { permanent_uuid: "8e4e220c8d154c4e9f4c68ad7a39efe7" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 38191 } }
I20260807 22:56:15.192641 28512 raft_consensus.cc:374] T ce237cf27e5a4781a47817430cc184ed P 8e4e220c8d154c4e9f4c68ad7a39efe7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:56:15.192807 28512 raft_consensus.cc:729] T ce237cf27e5a4781a47817430cc184ed P 8e4e220c8d154c4e9f4c68ad7a39efe7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e4e220c8d154c4e9f4c68ad7a39efe7, State: Initialized, Role: FOLLOWER
I20260807 22:56:15.193244 28512 consensus_queue.cc:260] T ce237cf27e5a4781a47817430cc184ed P 8e4e220c8d154c4e9f4c68ad7a39efe7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54b985a98334406b81decd685506beac" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 46797 } } peers { permanent_uuid: "001a80a933c846228fcb9e7611f26102" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 44545 } } peers { permanent_uuid: "8e4e220c8d154c4e9f4c68ad7a39efe7" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 38191 } }
I20260807 22:56:15.196647 28512 ts_tablet_manager.cc:1428] T ce237cf27e5a4781a47817430cc184ed P 8e4e220c8d154c4e9f4c68ad7a39efe7: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260807 22:56:15.198753 28552 ts_tablet_manager.cc:1918] T 24c39a4277894fac8184f75cd9da6cd4 P 54b985a98334406b81decd685506beac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 22:56:15.199108 28552 log.cc:1196] T 24c39a4277894fac8184f75cd9da6cd4 P 54b985a98334406b81decd685506beac: Deleting WAL directory at /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786143361779964-25976-0/minicluster-data/ts-0/wal/wals/24c39a4277894fac8184f75cd9da6cd4
I20260807 22:56:15.199776 28551 ts_tablet_manager.cc:1918] T 24c39a4277894fac8184f75cd9da6cd4 P 8e4e220c8d154c4e9f4c68ad7a39efe7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 22:56:15.200678 28506 tablet_bootstrap.cc:492] T ce237cf27e5a4781a47817430cc184ed P 001a80a933c846228fcb9e7611f26102: Bootstrap starting.
I20260807 22:56:15.201992 28552 ts_tablet_manager.cc:1939] T 24c39a4277894fac8184f75cd9da6cd4 P 54b985a98334406b81decd685506beac: Deleting consensus metadata
I20260807 22:56:15.202531 28551 log.cc:1196] T 24c39a4277894fac8184f75cd9da6cd4 P 8e4e220c8d154c4e9f4c68ad7a39efe7: Deleting WAL directory at /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786143361779964-25976-0/minicluster-data/ts-2/wal/wals/24c39a4277894fac8184f75cd9da6cd4
I20260807 22:56:15.203079 28506 tablet_bootstrap.cc:654] T ce237cf27e5a4781a47817430cc184ed P 001a80a933c846228fcb9e7611f26102: Neither blocks nor log segments found. Creating new log.
I20260807 22:56:15.203724 28551 ts_tablet_manager.cc:1939] T 24c39a4277894fac8184f75cd9da6cd4 P 8e4e220c8d154c4e9f4c68ad7a39efe7: Deleting consensus metadata
I20260807 22:56:15.204289 28553 tablet_replica.cc:330] T 24c39a4277894fac8184f75cd9da6cd4 P 001a80a933c846228fcb9e7611f26102: stopping tablet replica
I20260807 22:56:15.205035 28509 tablet_bootstrap.cc:492] T ce237cf27e5a4781a47817430cc184ed P 54b985a98334406b81decd685506beac: No bootstrap required, opened a new log
I20260807 22:56:15.205356 28509 ts_tablet_manager.cc:1397] T ce237cf27e5a4781a47817430cc184ed P 54b985a98334406b81decd685506beac: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.010s
I20260807 22:56:15.206390 28509 raft_consensus.cc:348] T ce237cf27e5a4781a47817430cc184ed P 54b985a98334406b81decd685506beac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54b985a98334406b81decd685506beac" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 46797 } } peers { permanent_uuid: "001a80a933c846228fcb9e7611f26102" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 44545 } } peers { permanent_uuid: "8e4e220c8d154c4e9f4c68ad7a39efe7" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 38191 } }
I20260807 22:56:15.206645 28509 raft_consensus.cc:374] T ce237cf27e5a4781a47817430cc184ed P 54b985a98334406b81decd685506beac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:56:15.206739 28509 raft_consensus.cc:729] T ce237cf27e5a4781a47817430cc184ed P 54b985a98334406b81decd685506beac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54b985a98334406b81decd685506beac, State: Initialized, Role: FOLLOWER
I20260807 22:56:15.207105 28509 consensus_queue.cc:260] T ce237cf27e5a4781a47817430cc184ed P 54b985a98334406b81decd685506beac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54b985a98334406b81decd685506beac" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 46797 } } peers { permanent_uuid: "001a80a933c846228fcb9e7611f26102" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 44545 } } peers { permanent_uuid: "8e4e220c8d154c4e9f4c68ad7a39efe7" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 38191 } }
I20260807 22:56:15.207813 28509 ts_tablet_manager.cc:1428] T ce237cf27e5a4781a47817430cc184ed P 54b985a98334406b81decd685506beac: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260807 22:56:15.209738 27716 catalog_manager.cc:4841] TS 54b985a98334406b81decd685506beac (127.25.94.1:46797): tablet 24c39a4277894fac8184f75cd9da6cd4 (table test-table [id=16a4d4571fa94e1abcb460ebe7523d4d]) successfully deleted
I20260807 22:56:15.205003 28553 raft_consensus.cc:2229] T 24c39a4277894fac8184f75cd9da6cd4 P 001a80a933c846228fcb9e7611f26102 [term 1 LEADER]: Raft consensus shutting down.
I20260807 22:56:15.216001 27719 catalog_manager.cc:4841] TS 8e4e220c8d154c4e9f4c68ad7a39efe7 (127.25.94.3:38191): tablet 24c39a4277894fac8184f75cd9da6cd4 (table test-table [id=16a4d4571fa94e1abcb460ebe7523d4d]) successfully deleted
I20260807 22:56:15.216600 28553 raft_consensus.cc:2258] T 24c39a4277894fac8184f75cd9da6cd4 P 001a80a933c846228fcb9e7611f26102 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 22:56:15.220018 28553 ts_tablet_manager.cc:1905] T 24c39a4277894fac8184f75cd9da6cd4 P 001a80a933c846228fcb9e7611f26102: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 22:56:15.224285 28506 tablet_bootstrap.cc:492] T ce237cf27e5a4781a47817430cc184ed P 001a80a933c846228fcb9e7611f26102: No bootstrap required, opened a new log
I20260807 22:56:15.224640 28506 ts_tablet_manager.cc:1397] T ce237cf27e5a4781a47817430cc184ed P 001a80a933c846228fcb9e7611f26102: Time spent bootstrapping tablet: real 0.024s user 0.014s sys 0.007s
I20260807 22:56:15.226267 28506 raft_consensus.cc:348] T ce237cf27e5a4781a47817430cc184ed P 001a80a933c846228fcb9e7611f26102 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54b985a98334406b81decd685506beac" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 46797 } } peers { permanent_uuid: "001a80a933c846228fcb9e7611f26102" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 44545 } } peers { permanent_uuid: "8e4e220c8d154c4e9f4c68ad7a39efe7" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 38191 } }
I20260807 22:56:15.226630 28506 raft_consensus.cc:374] T ce237cf27e5a4781a47817430cc184ed P 001a80a933c846228fcb9e7611f26102 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:56:15.226845 28506 raft_consensus.cc:729] T ce237cf27e5a4781a47817430cc184ed P 001a80a933c846228fcb9e7611f26102 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 001a80a933c846228fcb9e7611f26102, State: Initialized, Role: FOLLOWER
I20260807 22:56:15.227193 28506 consensus_queue.cc:260] T ce237cf27e5a4781a47817430cc184ed P 001a80a933c846228fcb9e7611f26102 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54b985a98334406b81decd685506beac" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 46797 } } peers { permanent_uuid: "001a80a933c846228fcb9e7611f26102" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 44545 } } peers { permanent_uuid: "8e4e220c8d154c4e9f4c68ad7a39efe7" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 38191 } }
I20260807 22:56:15.227917 28506 ts_tablet_manager.cc:1428] T ce237cf27e5a4781a47817430cc184ed P 001a80a933c846228fcb9e7611f26102: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260807 22:56:15.232369 28553 ts_tablet_manager.cc:1918] T 24c39a4277894fac8184f75cd9da6cd4 P 001a80a933c846228fcb9e7611f26102: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 22:56:15.232583 28553 log.cc:1196] T 24c39a4277894fac8184f75cd9da6cd4 P 001a80a933c846228fcb9e7611f26102: Deleting WAL directory at /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786143361779964-25976-0/minicluster-data/ts-1/wal/wals/24c39a4277894fac8184f75cd9da6cd4
I20260807 22:56:15.233492 28553 ts_tablet_manager.cc:1939] T 24c39a4277894fac8184f75cd9da6cd4 P 001a80a933c846228fcb9e7611f26102: Deleting consensus metadata
I20260807 22:56:15.235117 27715 catalog_manager.cc:4841] TS 001a80a933c846228fcb9e7611f26102 (127.25.94.2:44545): tablet 24c39a4277894fac8184f75cd9da6cd4 (table test-table [id=16a4d4571fa94e1abcb460ebe7523d4d]) successfully deleted
I20260807 22:56:15.492285 28519 raft_consensus.cc:482] T ce237cf27e5a4781a47817430cc184ed P 8e4e220c8d154c4e9f4c68ad7a39efe7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 22:56:15.492564 28519 raft_consensus.cc:504] T ce237cf27e5a4781a47817430cc184ed P 8e4e220c8d154c4e9f4c68ad7a39efe7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54b985a98334406b81decd685506beac" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 46797 } } peers { permanent_uuid: "001a80a933c846228fcb9e7611f26102" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 44545 } } peers { permanent_uuid: "8e4e220c8d154c4e9f4c68ad7a39efe7" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 38191 } }
I20260807 22:56:15.494199 28519 leader_election.cc:288] T ce237cf27e5a4781a47817430cc184ed P 8e4e220c8d154c4e9f4c68ad7a39efe7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 54b985a98334406b81decd685506beac (127.25.94.1:46797), 001a80a933c846228fcb9e7611f26102 (127.25.94.2:44545)
I20260807 22:56:15.502574 28503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:15.494961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.3:47679 (local address 127.25.94.1:46797)
0807 22:56:15.495163 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:15.495171 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:15.495555 (+ 384us) server_negotiation.cc:407] Connection header received
0807 22:56:15.495867 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:15.495872 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:15.495953 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:15.496005 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:56:15.496788 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:15.497742 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:15.500258 (+ 2516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:15.501467 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:15.501612 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:15.501727 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:15.502105 (+ 378us) server_negotiation.cc:299] Negotiation successful
0807 22:56:15.502277 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260807 22:56:15.502653 28561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:15.494481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.1:46797 (local address 127.25.94.3:47679)
0807 22:56:15.495417 (+ 936us) negotiation.cc:107] Waiting for socket to connect
0807 22:56:15.495440 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0807 22:56:15.495687 (+ 247us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:56:15.496219 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:56:15.496222 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:56:15.496242 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:56:15.496633 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:15.496657 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:15.497890 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:15.497898 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:15.500085 (+ 2187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:15.500100 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:15.501684 (+ 1584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:15.501691 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:15.501900 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:15.501937 (+ 37us) client_negotiation.cc:769] Sending connection context
0807 22:56:15.502050 (+ 113us) client_negotiation.cc:240] Negotiation successful
0807 22:56:15.502201 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":805,"mutex_wait_us":44,"thread_start_us":924,"threads_started":1}
I20260807 22:56:15.503424 28502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:15.494718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.3:35419 (local address 127.25.94.2:44545)
0807 22:56:15.494972 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:15.494981 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:15.495739 (+ 758us) server_negotiation.cc:407] Connection header received
0807 22:56:15.495863 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:15.495870 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:15.495977 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:15.496085 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:56:15.496756 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:15.498085 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:15.500754 (+ 2669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:15.502133 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:15.502294 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:15.502395 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:15.503024 (+ 629us) server_negotiation.cc:299] Negotiation successful
0807 22:56:15.503186 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260807 22:56:15.503716 27982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce237cf27e5a4781a47817430cc184ed" candidate_uuid: "8e4e220c8d154c4e9f4c68ad7a39efe7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54b985a98334406b81decd685506beac" is_pre_election: true
I20260807 22:56:15.504010 27982 raft_consensus.cc:2454] T ce237cf27e5a4781a47817430cc184ed P 54b985a98334406b81decd685506beac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e4e220c8d154c4e9f4c68ad7a39efe7 in term 0.
I20260807 22:56:15.504077 28559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:15.494512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.2:44545 (local address 127.25.94.3:35419)
0807 22:56:15.495626 (+ 1114us) negotiation.cc:107] Waiting for socket to connect
0807 22:56:15.495643 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 22:56:15.495744 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:56:15.496177 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:56:15.496182 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:56:15.496212 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:56:15.496565 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:15.496582 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:15.498313 (+ 1731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:15.498319 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:15.500548 (+ 2229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:15.500565 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:15.502585 (+ 2020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:15.502593 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:15.502836 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:15.502867 (+ 31us) client_negotiation.cc:769] Sending connection context
0807 22:56:15.503505 (+ 638us) client_negotiation.cc:240] Negotiation successful
0807 22:56:15.503669 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1024,"mutex_wait_us":62,"spinlock_wait_cycles":11520,"thread_start_us":621,"threads_started":1}
I20260807 22:56:15.504835 28303 leader_election.cc:302] T ce237cf27e5a4781a47817430cc184ed P 8e4e220c8d154c4e9f4c68ad7a39efe7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54b985a98334406b81decd685506beac, 8e4e220c8d154c4e9f4c68ad7a39efe7; no voters:
I20260807 22:56:15.505487 28519 raft_consensus.cc:2781] T ce237cf27e5a4781a47817430cc184ed P 8e4e220c8d154c4e9f4c68ad7a39efe7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 22:56:15.505760 28519 raft_consensus.cc:482] T ce237cf27e5a4781a47817430cc184ed P 8e4e220c8d154c4e9f4c68ad7a39efe7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 22:56:15.505986 28519 raft_consensus.cc:3037] T ce237cf27e5a4781a47817430cc184ed P 8e4e220c8d154c4e9f4c68ad7a39efe7 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:56:15.505671 28190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce237cf27e5a4781a47817430cc184ed" candidate_uuid: "8e4e220c8d154c4e9f4c68ad7a39efe7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "001a80a933c846228fcb9e7611f26102" is_pre_election: true
I20260807 22:56:15.506317 28190 raft_consensus.cc:2454] T ce237cf27e5a4781a47817430cc184ed P 001a80a933c846228fcb9e7611f26102 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e4e220c8d154c4e9f4c68ad7a39efe7 in term 0.
I20260807 22:56:15.508669 28519 raft_consensus.cc:504] T ce237cf27e5a4781a47817430cc184ed P 8e4e220c8d154c4e9f4c68ad7a39efe7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54b985a98334406b81decd685506beac" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 46797 } } peers { permanent_uuid: "001a80a933c846228fcb9e7611f26102" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 44545 } } peers { permanent_uuid: "8e4e220c8d154c4e9f4c68ad7a39efe7" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 38191 } }
I20260807 22:56:15.509420 28519 leader_election.cc:288] T ce237cf27e5a4781a47817430cc184ed P 8e4e220c8d154c4e9f4c68ad7a39efe7 [CANDIDATE]: Term 1 election: Requested vote from peers 54b985a98334406b81decd685506beac (127.25.94.1:46797), 001a80a933c846228fcb9e7611f26102 (127.25.94.2:44545)
I20260807 22:56:15.509940 27982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce237cf27e5a4781a47817430cc184ed" candidate_uuid: "8e4e220c8d154c4e9f4c68ad7a39efe7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54b985a98334406b81decd685506beac"
I20260807 22:56:15.510042 28190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce237cf27e5a4781a47817430cc184ed" candidate_uuid: "8e4e220c8d154c4e9f4c68ad7a39efe7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "001a80a933c846228fcb9e7611f26102"
I20260807 22:56:15.510166 27982 raft_consensus.cc:3037] T ce237cf27e5a4781a47817430cc184ed P 54b985a98334406b81decd685506beac [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:56:15.510341 28190 raft_consensus.cc:3037] T ce237cf27e5a4781a47817430cc184ed P 001a80a933c846228fcb9e7611f26102 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:56:15.512429 27982 raft_consensus.cc:2454] T ce237cf27e5a4781a47817430cc184ed P 54b985a98334406b81decd685506beac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e4e220c8d154c4e9f4c68ad7a39efe7 in term 1.
I20260807 22:56:15.512871 28303 leader_election.cc:302] T ce237cf27e5a4781a47817430cc184ed P 8e4e220c8d154c4e9f4c68ad7a39efe7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54b985a98334406b81decd685506beac, 8e4e220c8d154c4e9f4c68ad7a39efe7; no voters:
I20260807 22:56:15.513206 28519 raft_consensus.cc:2781] T ce237cf27e5a4781a47817430cc184ed P 8e4e220c8d154c4e9f4c68ad7a39efe7 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:56:15.514010 28519 raft_consensus.cc:686] T ce237cf27e5a4781a47817430cc184ed P 8e4e220c8d154c4e9f4c68ad7a39efe7 [term 1 LEADER]: Becoming Leader. State: Replica: 8e4e220c8d154c4e9f4c68ad7a39efe7, State: Running, Role: LEADER
I20260807 22:56:15.514537 28519 consensus_queue.cc:237] T ce237cf27e5a4781a47817430cc184ed P 8e4e220c8d154c4e9f4c68ad7a39efe7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54b985a98334406b81decd685506beac" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 46797 } } peers { permanent_uuid: "001a80a933c846228fcb9e7611f26102" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 44545 } } peers { permanent_uuid: "8e4e220c8d154c4e9f4c68ad7a39efe7" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 38191 } }
I20260807 22:56:15.514932 28190 raft_consensus.cc:2454] T ce237cf27e5a4781a47817430cc184ed P 001a80a933c846228fcb9e7611f26102 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e4e220c8d154c4e9f4c68ad7a39efe7 in term 1.
I20260807 22:56:15.520800 27739 catalog_manager.cc:5495] T ce237cf27e5a4781a47817430cc184ed P 8e4e220c8d154c4e9f4c68ad7a39efe7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8e4e220c8d154c4e9f4c68ad7a39efe7 (127.25.94.3). New cstate: current_term: 1 leader_uuid: "8e4e220c8d154c4e9f4c68ad7a39efe7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54b985a98334406b81decd685506beac" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 46797 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "001a80a933c846228fcb9e7611f26102" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 44545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e4e220c8d154c4e9f4c68ad7a39efe7" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 38191 } health_report { overall_health: HEALTHY } } }
Aug 07 22:56:15 dist-test-slave-z8x9 krb5kdc[27673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143373, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.62@KRBTEST.COM for kudu/127.25.94.62@KRBTEST.COM
I20260807 22:56:15.627178 28575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:15.611572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:43863 (local address 127.0.0.1:59228)
0807 22:56:15.612265 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0807 22:56:15.612303 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0807 22:56:15.612487 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:56:15.612889 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:56:15.612896 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:56:15.613628 (+ 732us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:56:15.613833 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:15.613843 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:15.615080 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:15.615085 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:15.616022 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:15.616033 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:15.617067 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:15.617073 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:15.617306 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:15.618224 (+ 918us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:56:15.618240 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:56:15.620972 (+ 2732us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:56:15.624196 (+ 3224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:56:15.624210 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:56:15.624218 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:56:15.624647 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:56:15.625020 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:56:15.625026 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:56:15.625029 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:56:15.625231 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:56:15.625863 (+ 632us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:56:15.625871 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:56:15.626047 (+ 176us) client_negotiation.cc:769] Sending connection context
0807 22:56:15.626234 (+ 187us) client_negotiation.cc:240] Negotiation successful
0807 22:56:15.626578 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":368,"threads_started":1}
I20260807 22:56:15.627301 28578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:15.611608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59228 (local address 127.25.94.62:43863)
0807 22:56:15.612627 (+ 1019us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:15.612631 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:15.612647 (+ 16us) server_negotiation.cc:407] Connection header received
0807 22:56:15.612686 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:15.612689 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:15.612740 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:15.612854 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:56:15.613993 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:15.614928 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:15.616170 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:15.616888 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:15.617047 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:15.621245 (+ 4198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:56:15.621255 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:56:15.621264 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:56:15.621293 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:56:15.624059 (+ 2766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:15.624777 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:15.624783 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:15.624792 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:15.624910 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:15.625349 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:15.625356 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:15.625359 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:15.625722 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:56:15.625887 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:15.626371 (+ 484us) server_negotiation.cc:299] Negotiation successful
0807 22:56:15.626712 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":136,"server-negotiator.queue_time_us":873,"thread_start_us":728,"threads_started":1}
I20260807 22:56:15.633841 27739 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.25.94.62@KRBTEST.COM'} at 127.0.0.1:59228:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260807 22:56:15.635270 27739 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 22:56:15.643246 28163 tablet_service.cc:1467] Processing CreateTablet for tablet ae36043fd04e45f799d919e491dbec65 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1bb65072195b490fa0af6cf50e3f07fe]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260807 22:56:15.643325 27941 tablet_service.cc:1467] Processing CreateTablet for tablet ae36043fd04e45f799d919e491dbec65 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1bb65072195b490fa0af6cf50e3f07fe]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260807 22:56:15.643951 28163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae36043fd04e45f799d919e491dbec65. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:56:15.643955 27941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae36043fd04e45f799d919e491dbec65. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:56:15.644753 28369 tablet_service.cc:1467] Processing CreateTablet for tablet ae36043fd04e45f799d919e491dbec65 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1bb65072195b490fa0af6cf50e3f07fe]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260807 22:56:15.645565 28369 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae36043fd04e45f799d919e491dbec65. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:56:15.650835 28509 tablet_bootstrap.cc:492] T ae36043fd04e45f799d919e491dbec65 P 54b985a98334406b81decd685506beac: Bootstrap starting.
I20260807 22:56:15.652781 28506 tablet_bootstrap.cc:492] T ae36043fd04e45f799d919e491dbec65 P 001a80a933c846228fcb9e7611f26102: Bootstrap starting.
I20260807 22:56:15.652911 28509 tablet_bootstrap.cc:654] T ae36043fd04e45f799d919e491dbec65 P 54b985a98334406b81decd685506beac: Neither blocks nor log segments found. Creating new log.
I20260807 22:56:15.654420 28512 tablet_bootstrap.cc:492] T ae36043fd04e45f799d919e491dbec65 P 8e4e220c8d154c4e9f4c68ad7a39efe7: Bootstrap starting.
I20260807 22:56:15.654771 28506 tablet_bootstrap.cc:654] T ae36043fd04e45f799d919e491dbec65 P 001a80a933c846228fcb9e7611f26102: Neither blocks nor log segments found. Creating new log.
I20260807 22:56:15.656638 28512 tablet_bootstrap.cc:654] T ae36043fd04e45f799d919e491dbec65 P 8e4e220c8d154c4e9f4c68ad7a39efe7: Neither blocks nor log segments found. Creating new log.
I20260807 22:56:15.656931 28509 tablet_bootstrap.cc:492] T ae36043fd04e45f799d919e491dbec65 P 54b985a98334406b81decd685506beac: No bootstrap required, opened a new log
I20260807 22:56:15.657116 28509 ts_tablet_manager.cc:1397] T ae36043fd04e45f799d919e491dbec65 P 54b985a98334406b81decd685506beac: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260807 22:56:15.657442 28506 tablet_bootstrap.cc:492] T ae36043fd04e45f799d919e491dbec65 P 001a80a933c846228fcb9e7611f26102: No bootstrap required, opened a new log
I20260807 22:56:15.657599 28506 ts_tablet_manager.cc:1397] T ae36043fd04e45f799d919e491dbec65 P 001a80a933c846228fcb9e7611f26102: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260807 22:56:15.657977 28509 raft_consensus.cc:348] T ae36043fd04e45f799d919e491dbec65 P 54b985a98334406b81decd685506beac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54b985a98334406b81decd685506beac" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 46797 } } peers { permanent_uuid: "001a80a933c846228fcb9e7611f26102" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 44545 } } peers { permanent_uuid: "8e4e220c8d154c4e9f4c68ad7a39efe7" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 38191 } }
I20260807 22:56:15.658224 28509 raft_consensus.cc:374] T ae36043fd04e45f799d919e491dbec65 P 54b985a98334406b81decd685506beac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:56:15.658322 28509 raft_consensus.cc:729] T ae36043fd04e45f799d919e491dbec65 P 54b985a98334406b81decd685506beac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54b985a98334406b81decd685506beac, State: Initialized, Role: FOLLOWER
I20260807 22:56:15.658372 28506 raft_consensus.cc:348] T ae36043fd04e45f799d919e491dbec65 P 001a80a933c846228fcb9e7611f26102 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54b985a98334406b81decd685506beac" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 46797 } } peers { permanent_uuid: "001a80a933c846228fcb9e7611f26102" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 44545 } } peers { permanent_uuid: "8e4e220c8d154c4e9f4c68ad7a39efe7" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 38191 } }
I20260807 22:56:15.658602 28506 raft_consensus.cc:374] T ae36043fd04e45f799d919e491dbec65 P 001a80a933c846228fcb9e7611f26102 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:56:15.658726 28512 tablet_bootstrap.cc:492] T ae36043fd04e45f799d919e491dbec65 P 8e4e220c8d154c4e9f4c68ad7a39efe7: No bootstrap required, opened a new log
I20260807 22:56:15.658592 28509 consensus_queue.cc:260] T ae36043fd04e45f799d919e491dbec65 P 54b985a98334406b81decd685506beac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54b985a98334406b81decd685506beac" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 46797 } } peers { permanent_uuid: "001a80a933c846228fcb9e7611f26102" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 44545 } } peers { permanent_uuid: "8e4e220c8d154c4e9f4c68ad7a39efe7" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 38191 } }
I20260807 22:56:15.658735 28506 raft_consensus.cc:729] T ae36043fd04e45f799d919e491dbec65 P 001a80a933c846228fcb9e7611f26102 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 001a80a933c846228fcb9e7611f26102, State: Initialized, Role: FOLLOWER
I20260807 22:56:15.659271 28509 ts_tablet_manager.cc:1428] T ae36043fd04e45f799d919e491dbec65 P 54b985a98334406b81decd685506beac: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260807 22:56:15.659180 28520 tablet_replica.cc:439] T ae36043fd04e45f799d919e491dbec65 P 54b985a98334406b81decd685506beac: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54b985a98334406b81decd685506beac" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 46797 } } peers { permanent_uuid: "001a80a933c846228fcb9e7611f26102" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 44545 } } peers { permanent_uuid: "8e4e220c8d154c4e9f4c68ad7a39efe7" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 38191 } } }
I20260807 22:56:15.659564 28520 tablet_replica.cc:442] T ae36043fd04e45f799d919e491dbec65 P 54b985a98334406b81decd685506beac: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 22:56:15.659716 28506 consensus_queue.cc:260] T ae36043fd04e45f799d919e491dbec65 P 001a80a933c846228fcb9e7611f26102 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54b985a98334406b81decd685506beac" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 46797 } } peers { permanent_uuid: "001a80a933c846228fcb9e7611f26102" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 44545 } } peers { permanent_uuid: "8e4e220c8d154c4e9f4c68ad7a39efe7" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 38191 } }
I20260807 22:56:15.660198 28542 tablet_replica.cc:439] T ae36043fd04e45f799d919e491dbec65 P 001a80a933c846228fcb9e7611f26102: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54b985a98334406b81decd685506beac" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 46797 } } peers { permanent_uuid: "001a80a933c846228fcb9e7611f26102" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 44545 } } peers { permanent_uuid: "8e4e220c8d154c4e9f4c68ad7a39efe7" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 38191 } } }
I20260807 22:56:15.658953 28512 ts_tablet_manager.cc:1397] T ae36043fd04e45f799d919e491dbec65 P 8e4e220c8d154c4e9f4c68ad7a39efe7: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260807 22:56:15.660511 28542 tablet_replica.cc:442] T ae36043fd04e45f799d919e491dbec65 P 001a80a933c846228fcb9e7611f26102: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 22:56:15.661001 28506 ts_tablet_manager.cc:1428] T ae36043fd04e45f799d919e491dbec65 P 001a80a933c846228fcb9e7611f26102: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260807 22:56:15.661502 28512 raft_consensus.cc:348] T ae36043fd04e45f799d919e491dbec65 P 8e4e220c8d154c4e9f4c68ad7a39efe7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54b985a98334406b81decd685506beac" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 46797 } } peers { permanent_uuid: "001a80a933c846228fcb9e7611f26102" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 44545 } } peers { permanent_uuid: "8e4e220c8d154c4e9f4c68ad7a39efe7" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 38191 } }
I20260807 22:56:15.661825 28512 raft_consensus.cc:374] T ae36043fd04e45f799d919e491dbec65 P 8e4e220c8d154c4e9f4c68ad7a39efe7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:56:15.661923 28512 raft_consensus.cc:729] T ae36043fd04e45f799d919e491dbec65 P 8e4e220c8d154c4e9f4c68ad7a39efe7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e4e220c8d154c4e9f4c68ad7a39efe7, State: Initialized, Role: FOLLOWER
I20260807 22:56:15.662186 28512 consensus_queue.cc:260] T ae36043fd04e45f799d919e491dbec65 P 8e4e220c8d154c4e9f4c68ad7a39efe7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54b985a98334406b81decd685506beac" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 46797 } } peers { permanent_uuid: "001a80a933c846228fcb9e7611f26102" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 44545 } } peers { permanent_uuid: "8e4e220c8d154c4e9f4c68ad7a39efe7" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 38191 } }
I20260807 22:56:15.662665 28519 tablet_replica.cc:439] T ae36043fd04e45f799d919e491dbec65 P 8e4e220c8d154c4e9f4c68ad7a39efe7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54b985a98334406b81decd685506beac" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 46797 } } peers { permanent_uuid: "001a80a933c846228fcb9e7611f26102" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 44545 } } peers { permanent_uuid: "8e4e220c8d154c4e9f4c68ad7a39efe7" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 38191 } } }
I20260807 22:56:15.662968 28519 tablet_replica.cc:442] T ae36043fd04e45f799d919e491dbec65 P 8e4e220c8d154c4e9f4c68ad7a39efe7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 22:56:15.663477 28512 ts_tablet_manager.cc:1428] T ae36043fd04e45f799d919e491dbec65 P 8e4e220c8d154c4e9f4c68ad7a39efe7: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260807 22:56:15.901585 28520 raft_consensus.cc:482] T ae36043fd04e45f799d919e491dbec65 P 54b985a98334406b81decd685506beac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 22:56:15.901923 28520 raft_consensus.cc:504] T ae36043fd04e45f799d919e491dbec65 P 54b985a98334406b81decd685506beac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54b985a98334406b81decd685506beac" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 46797 } } peers { permanent_uuid: "001a80a933c846228fcb9e7611f26102" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 44545 } } peers { permanent_uuid: "8e4e220c8d154c4e9f4c68ad7a39efe7" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 38191 } }
I20260807 22:56:15.903648 28520 leader_election.cc:288] T ae36043fd04e45f799d919e491dbec65 P 54b985a98334406b81decd685506beac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 001a80a933c846228fcb9e7611f26102 (127.25.94.2:44545), 8e4e220c8d154c4e9f4c68ad7a39efe7 (127.25.94.3:38191)
I20260807 22:56:15.914984 28502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:15.904142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.1:48455 (local address 127.25.94.2:44545)
0807 22:56:15.904436 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:15.904446 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:15.906146 (+ 1700us) server_negotiation.cc:407] Connection header received
0807 22:56:15.906605 (+ 459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:15.906636 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:15.906746 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:15.906845 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:56:15.908082 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:15.909102 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:15.912421 (+ 3319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:15.913523 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:15.913689 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:15.913837 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:15.914503 (+ 666us) server_negotiation.cc:299] Negotiation successful
0807 22:56:15.914672 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125}
I20260807 22:56:15.915256 28582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:15.904069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.3:38191 (local address 127.25.94.1:39815)
0807 22:56:15.905154 (+ 1085us) negotiation.cc:107] Waiting for socket to connect
0807 22:56:15.905219 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0807 22:56:15.905442 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:56:15.906667 (+ 1225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:56:15.906672 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:56:15.906696 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:56:15.907094 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:15.907110 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:15.909219 (+ 2109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:15.909228 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:15.912087 (+ 2859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:15.912107 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:15.914031 (+ 1924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:15.914045 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:15.914357 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:15.914414 (+ 57us) client_negotiation.cc:769] Sending connection context
0807 22:56:15.914500 (+ 86us) client_negotiation.cc:240] Negotiation successful
0807 22:56:15.914696 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":812,"mutex_wait_us":59,"thread_start_us":2534,"threads_started":1}
I20260807 22:56:15.915735 28584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:15.904138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.2:44545 (local address 127.25.94.1:48455)
0807 22:56:15.906017 (+ 1879us) negotiation.cc:107] Waiting for socket to connect
0807 22:56:15.906036 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 22:56:15.906325 (+ 289us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:56:15.907173 (+ 848us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:56:15.907180 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:56:15.907209 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:56:15.907872 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:15.907897 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:15.909455 (+ 1558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:15.909461 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:15.912173 (+ 2712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:15.912190 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:15.914067 (+ 1877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:15.914075 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:15.914303 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:15.914351 (+ 48us) client_negotiation.cc:769] Sending connection context
0807 22:56:15.915177 (+ 826us) client_negotiation.cc:240] Negotiation successful
0807 22:56:15.915330 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1695,"mutex_wait_us":90,"thread_start_us":2620,"threads_started":1}
I20260807 22:56:15.916429 28583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:15.904111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.1:39815 (local address 127.25.94.3:38191)
0807 22:56:15.906298 (+ 2187us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:15.906304 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:15.906330 (+ 26us) server_negotiation.cc:407] Connection header received
0807 22:56:15.906418 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:15.906423 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:15.906496 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:15.907290 (+ 794us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:56:15.907601 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:15.909018 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:15.912336 (+ 3318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:15.913735 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:15.915548 (+ 1813us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:15.915721 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:15.915787 (+ 66us) server_negotiation.cc:299] Negotiation successful
0807 22:56:15.915979 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2034,"thread_start_us":1872,"threads_started":1}
I20260807 22:56:15.917084 28190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae36043fd04e45f799d919e491dbec65" candidate_uuid: "54b985a98334406b81decd685506beac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "001a80a933c846228fcb9e7611f26102" is_pre_election: true
I20260807 22:56:15.917271 28408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae36043fd04e45f799d919e491dbec65" candidate_uuid: "54b985a98334406b81decd685506beac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e4e220c8d154c4e9f4c68ad7a39efe7" is_pre_election: true
I20260807 22:56:15.917398 28190 raft_consensus.cc:2454] T ae36043fd04e45f799d919e491dbec65 P 001a80a933c846228fcb9e7611f26102 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54b985a98334406b81decd685506beac in term 0.
I20260807 22:56:15.917536 28408 raft_consensus.cc:2454] T ae36043fd04e45f799d919e491dbec65 P 8e4e220c8d154c4e9f4c68ad7a39efe7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54b985a98334406b81decd685506beac in term 0.
I20260807 22:56:15.918088 27872 leader_election.cc:302] T ae36043fd04e45f799d919e491dbec65 P 54b985a98334406b81decd685506beac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54b985a98334406b81decd685506beac, 8e4e220c8d154c4e9f4c68ad7a39efe7; no voters:
I20260807 22:56:15.918566 28520 raft_consensus.cc:2781] T ae36043fd04e45f799d919e491dbec65 P 54b985a98334406b81decd685506beac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 22:56:15.918727 28520 raft_consensus.cc:482] T ae36043fd04e45f799d919e491dbec65 P 54b985a98334406b81decd685506beac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 22:56:15.918859 28520 raft_consensus.cc:3037] T ae36043fd04e45f799d919e491dbec65 P 54b985a98334406b81decd685506beac [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:56:15.920994 28519 consensus_queue.cc:1035] T ce237cf27e5a4781a47817430cc184ed P 8e4e220c8d154c4e9f4c68ad7a39efe7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "001a80a933c846228fcb9e7611f26102" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 44545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 22:56:15.922405 28520 raft_consensus.cc:504] T ae36043fd04e45f799d919e491dbec65 P 54b985a98334406b81decd685506beac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54b985a98334406b81decd685506beac" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 46797 } } peers { permanent_uuid: "001a80a933c846228fcb9e7611f26102" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 44545 } } peers { permanent_uuid: "8e4e220c8d154c4e9f4c68ad7a39efe7" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 38191 } }
I20260807 22:56:15.923082 28520 leader_election.cc:288] T ae36043fd04e45f799d919e491dbec65 P 54b985a98334406b81decd685506beac [CANDIDATE]: Term 1 election: Requested vote from peers 001a80a933c846228fcb9e7611f26102 (127.25.94.2:44545), 8e4e220c8d154c4e9f4c68ad7a39efe7 (127.25.94.3:38191)
I20260807 22:56:15.923664 28189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae36043fd04e45f799d919e491dbec65" candidate_uuid: "54b985a98334406b81decd685506beac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "001a80a933c846228fcb9e7611f26102"
I20260807 22:56:15.923748 28408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae36043fd04e45f799d919e491dbec65" candidate_uuid: "54b985a98334406b81decd685506beac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e4e220c8d154c4e9f4c68ad7a39efe7"
I20260807 22:56:15.923988 28189 raft_consensus.cc:3037] T ae36043fd04e45f799d919e491dbec65 P 001a80a933c846228fcb9e7611f26102 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:56:15.924037 28408 raft_consensus.cc:3037] T ae36043fd04e45f799d919e491dbec65 P 8e4e220c8d154c4e9f4c68ad7a39efe7 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:56:15.927495 28189 raft_consensus.cc:2454] T ae36043fd04e45f799d919e491dbec65 P 001a80a933c846228fcb9e7611f26102 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54b985a98334406b81decd685506beac in term 1.
I20260807 22:56:15.927819 28408 raft_consensus.cc:2454] T ae36043fd04e45f799d919e491dbec65 P 8e4e220c8d154c4e9f4c68ad7a39efe7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54b985a98334406b81decd685506beac in term 1.
I20260807 22:56:15.928031 27867 leader_election.cc:302] T ae36043fd04e45f799d919e491dbec65 P 54b985a98334406b81decd685506beac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 001a80a933c846228fcb9e7611f26102, 54b985a98334406b81decd685506beac; no voters:
I20260807 22:56:15.928520 28520 raft_consensus.cc:2781] T ae36043fd04e45f799d919e491dbec65 P 54b985a98334406b81decd685506beac [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:56:15.928995 28520 raft_consensus.cc:686] T ae36043fd04e45f799d919e491dbec65 P 54b985a98334406b81decd685506beac [term 1 LEADER]: Becoming Leader. State: Replica: 54b985a98334406b81decd685506beac, State: Running, Role: LEADER
I20260807 22:56:15.929764 28520 consensus_queue.cc:237] T ae36043fd04e45f799d919e491dbec65 P 54b985a98334406b81decd685506beac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54b985a98334406b81decd685506beac" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 46797 } } peers { permanent_uuid: "001a80a933c846228fcb9e7611f26102" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 44545 } } peers { permanent_uuid: "8e4e220c8d154c4e9f4c68ad7a39efe7" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 38191 } }
I20260807 22:56:15.930621 28519 consensus_queue.cc:1035] T ce237cf27e5a4781a47817430cc184ed P 8e4e220c8d154c4e9f4c68ad7a39efe7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54b985a98334406b81decd685506beac" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 46797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 22:56:15.932317 28590 tablet_replica.cc:439] T ae36043fd04e45f799d919e491dbec65 P 54b985a98334406b81decd685506beac: TxnStatusTablet state changed. Reason: New leader 54b985a98334406b81decd685506beac. Latest consensus state: current_term: 1 leader_uuid: "54b985a98334406b81decd685506beac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54b985a98334406b81decd685506beac" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 46797 } } peers { permanent_uuid: "001a80a933c846228fcb9e7611f26102" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 44545 } } peers { permanent_uuid: "8e4e220c8d154c4e9f4c68ad7a39efe7" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 38191 } } }
I20260807 22:56:15.932765 28590 tablet_replica.cc:442] T ae36043fd04e45f799d919e491dbec65 P 54b985a98334406b81decd685506beac: This TxnStatusTablet replica's current role is: LEADER
I20260807 22:56:15.935391 27738 catalog_manager.cc:5495] T ae36043fd04e45f799d919e491dbec65 P 54b985a98334406b81decd685506beac reported cstate change: term changed from 0 to 1, leader changed from <none> to 54b985a98334406b81decd685506beac (127.25.94.1). New cstate: current_term: 1 leader_uuid: "54b985a98334406b81decd685506beac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54b985a98334406b81decd685506beac" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 46797 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "001a80a933c846228fcb9e7611f26102" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 44545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e4e220c8d154c4e9f4c68ad7a39efe7" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 38191 } health_report { overall_health: UNKNOWN } } }
I20260807 22:56:15.936081 28596 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260807 22:56:15.936462 28596 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260807 22:56:15.968340 28503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:15.957708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45294 (local address 127.25.94.1:46797)
0807 22:56:15.957930 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:15.957938 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:15.957962 (+ 24us) server_negotiation.cc:407] Connection header received
0807 22:56:15.958066 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:15.958072 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:15.958200 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:15.958358 (+ 158us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 22:56:15.958817 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:15.960190 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:15.962163 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:15.963062 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:15.963229 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:15.966124 (+ 2895us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 22:56:15.967001 (+ 877us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 22:56:15.967135 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:15.967918 (+ 783us) server_negotiation.cc:299] Negotiation successful
0807 22:56:15.968008 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260807 22:56:15.968377 28575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:15.956300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.1:46797 (local address 127.0.0.1:45294)
0807 22:56:15.956551 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0807 22:56:15.956581 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0807 22:56:15.956753 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:56:15.958387 (+ 1634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:56:15.958392 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:56:15.958405 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0807 22:56:15.958600 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:15.958612 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:15.960381 (+ 1769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:15.960388 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:15.962007 (+ 1619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:15.962019 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:15.963261 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:15.963267 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:15.963459 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:15.963544 (+ 85us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0807 22:56:15.967752 (+ 4208us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0807 22:56:15.967782 (+ 30us) client_negotiation.cc:769] Sending connection context
0807 22:56:15.967892 (+ 110us) client_negotiation.cc:240] Negotiation successful
0807 22:56:15.968059 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260807 22:56:15.973709 28190 raft_consensus.cc:1261] T ae36043fd04e45f799d919e491dbec65 P 001a80a933c846228fcb9e7611f26102 [term 1 FOLLOWER]: Refusing update from remote peer 54b985a98334406b81decd685506beac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 22:56:15.974723 28408 raft_consensus.cc:1261] T ae36043fd04e45f799d919e491dbec65 P 8e4e220c8d154c4e9f4c68ad7a39efe7 [term 1 FOLLOWER]: Refusing update from remote peer 54b985a98334406b81decd685506beac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 22:56:15.974929 28595 consensus_queue.cc:1035] T ae36043fd04e45f799d919e491dbec65 P 54b985a98334406b81decd685506beac [LEADER]: Connected to new peer: Peer: permanent_uuid: "001a80a933c846228fcb9e7611f26102" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 44545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 22:56:15.975493 28520 consensus_queue.cc:1035] T ae36043fd04e45f799d919e491dbec65 P 54b985a98334406b81decd685506beac [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e4e220c8d154c4e9f4c68ad7a39efe7" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 38191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 22:56:15.978541 28542 tablet_replica.cc:439] T ae36043fd04e45f799d919e491dbec65 P 001a80a933c846228fcb9e7611f26102: TxnStatusTablet state changed. Reason: New leader 54b985a98334406b81decd685506beac. Latest consensus state: current_term: 1 leader_uuid: "54b985a98334406b81decd685506beac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54b985a98334406b81decd685506beac" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 46797 } } peers { permanent_uuid: "001a80a933c846228fcb9e7611f26102" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 44545 } } peers { permanent_uuid: "8e4e220c8d154c4e9f4c68ad7a39efe7" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 38191 } } }
I20260807 22:56:15.978859 28542 tablet_replica.cc:442] T ae36043fd04e45f799d919e491dbec65 P 001a80a933c846228fcb9e7611f26102: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 22:56:15.980170 28519 tablet_replica.cc:439] T ae36043fd04e45f799d919e491dbec65 P 8e4e220c8d154c4e9f4c68ad7a39efe7: TxnStatusTablet state changed. Reason: New leader 54b985a98334406b81decd685506beac. Latest consensus state: current_term: 1 leader_uuid: "54b985a98334406b81decd685506beac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54b985a98334406b81decd685506beac" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 46797 } } peers { permanent_uuid: "001a80a933c846228fcb9e7611f26102" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 44545 } } peers { permanent_uuid: "8e4e220c8d154c4e9f4c68ad7a39efe7" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 38191 } } }
I20260807 22:56:15.980489 28519 tablet_replica.cc:442] T ae36043fd04e45f799d919e491dbec65 P 8e4e220c8d154c4e9f4c68ad7a39efe7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 22:56:15.984936 28590 tablet_replica.cc:439] T ae36043fd04e45f799d919e491dbec65 P 54b985a98334406b81decd685506beac: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "54b985a98334406b81decd685506beac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54b985a98334406b81decd685506beac" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 46797 } } peers { permanent_uuid: "001a80a933c846228fcb9e7611f26102" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 44545 } } peers { permanent_uuid: "8e4e220c8d154c4e9f4c68ad7a39efe7" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 38191 } } }
I20260807 22:56:15.986074 28590 tablet_replica.cc:442] T ae36043fd04e45f799d919e491dbec65 P 54b985a98334406b81decd685506beac: This TxnStatusTablet replica's current role is: LEADER
I20260807 22:56:15.988354 28587 mvcc.cc:205] Tried to move back new op lower bound from 7316043267980513280 to 7316043267813580800. Current Snapshot: MvccSnapshot[applied={T|T < 7316043267980513280 or (T in {7316043267980513280})}]
I20260807 22:56:15.990075 28519 tablet_replica.cc:439] T ae36043fd04e45f799d919e491dbec65 P 8e4e220c8d154c4e9f4c68ad7a39efe7: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "54b985a98334406b81decd685506beac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54b985a98334406b81decd685506beac" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 46797 } } peers { permanent_uuid: "001a80a933c846228fcb9e7611f26102" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 44545 } } peers { permanent_uuid: "8e4e220c8d154c4e9f4c68ad7a39efe7" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 38191 } } }
I20260807 22:56:15.990372 28519 tablet_replica.cc:442] T ae36043fd04e45f799d919e491dbec65 P 8e4e220c8d154c4e9f4c68ad7a39efe7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 22:56:15.991765 28602 mvcc.cc:205] Tried to move back new op lower bound from 7316043267980513280 to 7316043267813580800. Current Snapshot: MvccSnapshot[applied={T|T < 7316043267980513280 or (T in {7316043267980513280})}]
I20260807 22:56:15.995555 28589 mvcc.cc:205] Tried to move back new op lower bound from 7316043267980513280 to 7316043267813580800. Current Snapshot: MvccSnapshot[applied={T|T < 7316043267980513280}]
I20260807 22:56:15.997377 28520 tablet_replica.cc:439] T ae36043fd04e45f799d919e491dbec65 P 54b985a98334406b81decd685506beac: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "54b985a98334406b81decd685506beac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54b985a98334406b81decd685506beac" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 46797 } } peers { permanent_uuid: "001a80a933c846228fcb9e7611f26102" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 44545 } } peers { permanent_uuid: "8e4e220c8d154c4e9f4c68ad7a39efe7" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 38191 } } }
I20260807 22:56:15.998001 28520 tablet_replica.cc:442] T ae36043fd04e45f799d919e491dbec65 P 54b985a98334406b81decd685506beac: This TxnStatusTablet replica's current role is: LEADER
I20260807 22:56:15.999678 28542 tablet_replica.cc:439] T ae36043fd04e45f799d919e491dbec65 P 001a80a933c846228fcb9e7611f26102: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "54b985a98334406b81decd685506beac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54b985a98334406b81decd685506beac" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 46797 } } peers { permanent_uuid: "001a80a933c846228fcb9e7611f26102" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 44545 } } peers { permanent_uuid: "8e4e220c8d154c4e9f4c68ad7a39efe7" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 38191 } } }
I20260807 22:56:16.000172 28542 tablet_replica.cc:442] T ae36043fd04e45f799d919e491dbec65 P 001a80a933c846228fcb9e7611f26102: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 22:56:16.014998 28583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:15.994612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56682 (local address 127.25.94.3:38191)
0807 22:56:15.994824 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:15.994832 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:16.008823 (+ 13991us) server_negotiation.cc:407] Connection header received
0807 22:56:16.008999 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:16.009004 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:16.009136 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:16.009353 (+ 217us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 22:56:16.009901 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:16.011000 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:16.012448 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:16.013078 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:16.013247 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:16.013665 (+ 418us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 22:56:16.014413 (+ 748us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 22:56:16.014561 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:16.014615 (+ 54us) server_negotiation.cc:299] Negotiation successful
0807 22:56:16.014704 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260807 22:56:16.032840 28503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:16.026186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.3:45299 (local address 127.25.94.1:46797)
0807 22:56:16.026439 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:16.026444 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:16.027431 (+ 987us) server_negotiation.cc:407] Connection header received
0807 22:56:16.027688 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:16.027693 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:16.027774 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:16.027904 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 22:56:16.028475 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:16.029342 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:16.030896 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:16.031422 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:16.031603 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:16.031986 (+ 383us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 22:56:16.032104 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 22:56:16.032202 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:16.032364 (+ 162us) server_negotiation.cc:299] Negotiation successful
0807 22:56:16.032449 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260807 22:56:16.032949 28613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:16.026080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.1:46797 (local address 127.25.94.3:45299)
0807 22:56:16.027311 (+ 1231us) negotiation.cc:107] Waiting for socket to connect
0807 22:56:16.027329 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 22:56:16.027537 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:56:16.027918 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:56:16.027923 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:56:16.027934 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0807 22:56:16.028312 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:16.028322 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:16.029481 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:16.029488 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:16.030754 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:16.030766 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:16.031586 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:16.031593 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:16.031747 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:16.031804 (+ 57us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0807 22:56:16.032205 (+ 401us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0807 22:56:16.032219 (+ 14us) client_negotiation.cc:769] Sending connection context
0807 22:56:16.032324 (+ 105us) client_negotiation.cc:240] Negotiation successful
0807 22:56:16.032465 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1008,"thread_start_us":807,"threads_started":1}
I20260807 22:56:16.084964 28583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:16.076482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.1:59813 (local address 127.25.94.3:38191)
0807 22:56:16.076803 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:16.076896 (+ 93us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:16.078290 (+ 1394us) server_negotiation.cc:407] Connection header received
0807 22:56:16.078511 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:16.078517 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:16.078603 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:16.078702 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 22:56:16.079304 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:16.080313 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:16.082630 (+ 2317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:16.083314 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:16.083464 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:16.084082 (+ 618us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 22:56:16.084236 (+ 154us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 22:56:16.084344 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:16.084516 (+ 172us) server_negotiation.cc:299] Negotiation successful
0807 22:56:16.084611 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260807 22:56:16.085994 28623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:16.076037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.3:38191 (local address 127.25.94.1:59813)
0807 22:56:16.078141 (+ 2104us) negotiation.cc:107] Waiting for socket to connect
0807 22:56:16.078185 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0807 22:56:16.078356 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:56:16.078725 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:56:16.078729 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:56:16.078741 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0807 22:56:16.079146 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:16.079155 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:16.080514 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:16.080520 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:16.082451 (+ 1931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:16.082473 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:16.083587 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:16.083597 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:16.083797 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:16.083887 (+ 90us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0807 22:56:16.084371 (+ 484us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0807 22:56:16.084383 (+ 12us) client_negotiation.cc:769] Sending connection context
0807 22:56:16.084466 (+ 83us) client_negotiation.cc:240] Negotiation successful
0807 22:56:16.084592 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1564,"thread_start_us":1391,"threads_started":1}
I20260807 22:56:16.142637 27738 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59224:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 22:56:16.142925 27738 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59224:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 22:56:16.146896 27738 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 00842aecda594fed8f1d657ee997b2c3: Sending DeleteTablet for 3 replicas of tablet ce237cf27e5a4781a47817430cc184ed
I20260807 22:56:16.147897 27941 tablet_service.cc:1514] Processing DeleteTablet for tablet ce237cf27e5a4781a47817430cc184ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:56:16 UTC) from {username='kudu', principal='kudu/127.25.94.62@KRBTEST.COM'} at 127.0.0.1:45286
I20260807 22:56:16.148337 28163 tablet_service.cc:1514] Processing DeleteTablet for tablet ce237cf27e5a4781a47817430cc184ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:56:16 UTC) from {username='kudu', principal='kudu/127.25.94.62@KRBTEST.COM'} at 127.0.0.1:45538
I20260807 22:56:16.149246 28369 tablet_service.cc:1514] Processing DeleteTablet for tablet ce237cf27e5a4781a47817430cc184ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:56:16 UTC) from {username='kudu', principal='kudu/127.25.94.62@KRBTEST.COM'} at 127.0.0.1:56680
I20260807 22:56:16.176047 28628 tablet_replica.cc:330] T ce237cf27e5a4781a47817430cc184ed P 54b985a98334406b81decd685506beac: stopping tablet replica
I20260807 22:56:16.176795 28628 raft_consensus.cc:2229] T ce237cf27e5a4781a47817430cc184ed P 54b985a98334406b81decd685506beac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 22:56:16.177059 28630 tablet_replica.cc:330] T ce237cf27e5a4781a47817430cc184ed P 8e4e220c8d154c4e9f4c68ad7a39efe7: stopping tablet replica
I20260807 22:56:16.177351 28630 raft_consensus.cc:2229] T ce237cf27e5a4781a47817430cc184ed P 8e4e220c8d154c4e9f4c68ad7a39efe7 [term 1 LEADER]: Raft consensus shutting down.
I20260807 22:56:16.177369 28628 raft_consensus.cc:2258] T ce237cf27e5a4781a47817430cc184ed P 54b985a98334406b81decd685506beac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 22:56:16.178069 28630 raft_consensus.cc:2258] T ce237cf27e5a4781a47817430cc184ed P 8e4e220c8d154c4e9f4c68ad7a39efe7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 22:56:16.178701 28629 tablet_replica.cc:330] T ce237cf27e5a4781a47817430cc184ed P 001a80a933c846228fcb9e7611f26102: stopping tablet replica
I20260807 22:56:16.178936 28629 raft_consensus.cc:2229] T ce237cf27e5a4781a47817430cc184ed P 001a80a933c846228fcb9e7611f26102 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 22:56:16.178944 28628 ts_tablet_manager.cc:1905] T ce237cf27e5a4781a47817430cc184ed P 54b985a98334406b81decd685506beac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 22:56:16.179196 28629 raft_consensus.cc:2258] T ce237cf27e5a4781a47817430cc184ed P 001a80a933c846228fcb9e7611f26102 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 22:56:16.181522 28630 ts_tablet_manager.cc:1905] T ce237cf27e5a4781a47817430cc184ed P 8e4e220c8d154c4e9f4c68ad7a39efe7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 22:56:16.185449 28629 ts_tablet_manager.cc:1905] T ce237cf27e5a4781a47817430cc184ed P 001a80a933c846228fcb9e7611f26102: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 22:56:16.191167 28628 ts_tablet_manager.cc:1918] T ce237cf27e5a4781a47817430cc184ed P 54b985a98334406b81decd685506beac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 22:56:16.191509 28628 log.cc:1196] T ce237cf27e5a4781a47817430cc184ed P 54b985a98334406b81decd685506beac: Deleting WAL directory at /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786143361779964-25976-0/minicluster-data/ts-0/wal/wals/ce237cf27e5a4781a47817430cc184ed
I20260807 22:56:16.192162 28628 ts_tablet_manager.cc:1939] T ce237cf27e5a4781a47817430cc184ed P 54b985a98334406b81decd685506beac: Deleting consensus metadata
I20260807 22:56:16.193645 27716 catalog_manager.cc:4841] TS 54b985a98334406b81decd685506beac (127.25.94.1:46797): tablet ce237cf27e5a4781a47817430cc184ed (table test-table [id=4a7eef5298dc48d59cfec10fa50546e3]) successfully deleted
I20260807 22:56:16.198190 28630 ts_tablet_manager.cc:1918] T ce237cf27e5a4781a47817430cc184ed P 8e4e220c8d154c4e9f4c68ad7a39efe7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 22:56:16.198393 28630 log.cc:1196] T ce237cf27e5a4781a47817430cc184ed P 8e4e220c8d154c4e9f4c68ad7a39efe7: Deleting WAL directory at /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786143361779964-25976-0/minicluster-data/ts-2/wal/wals/ce237cf27e5a4781a47817430cc184ed
I20260807 22:56:16.198894 28630 ts_tablet_manager.cc:1939] T ce237cf27e5a4781a47817430cc184ed P 8e4e220c8d154c4e9f4c68ad7a39efe7: Deleting consensus metadata
I20260807 22:56:16.198925 28629 ts_tablet_manager.cc:1918] T ce237cf27e5a4781a47817430cc184ed P 001a80a933c846228fcb9e7611f26102: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 22:56:16.199208 28629 log.cc:1196] T ce237cf27e5a4781a47817430cc184ed P 001a80a933c846228fcb9e7611f26102: Deleting WAL directory at /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786143361779964-25976-0/minicluster-data/ts-1/wal/wals/ce237cf27e5a4781a47817430cc184ed
I20260807 22:56:16.199970 28629 ts_tablet_manager.cc:1939] T ce237cf27e5a4781a47817430cc184ed P 001a80a933c846228fcb9e7611f26102: Deleting consensus metadata
I20260807 22:56:16.200245 27719 catalog_manager.cc:4841] TS 8e4e220c8d154c4e9f4c68ad7a39efe7 (127.25.94.3:38191): tablet ce237cf27e5a4781a47817430cc184ed (table test-table [id=4a7eef5298dc48d59cfec10fa50546e3]) successfully deleted
I20260807 22:56:16.201432 27715 catalog_manager.cc:4841] TS 001a80a933c846228fcb9e7611f26102 (127.25.94.2:44545): tablet ce237cf27e5a4781a47817430cc184ed (table test-table [id=4a7eef5298dc48d59cfec10fa50546e3]) successfully deleted
Aug 07 22:56:16 dist-test-slave-z8x9 krb5kdc[27673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143374, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.25.94.1@KRBTEST.COM
I20260807 22:56:16.217013 28503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:16.175147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45298 (local address 127.25.94.1:46797)
0807 22:56:16.175406 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:16.175416 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:16.181901 (+ 6485us) server_negotiation.cc:407] Connection header received
0807 22:56:16.193617 (+ 11716us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:16.193623 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:16.193708 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:16.193781 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:56:16.196276 (+ 2495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:16.197909 (+ 1633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:16.200010 (+ 2101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:16.200699 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:16.200845 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:16.207750 (+ 6905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:56:16.207775 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:56:16.207828 (+ 53us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:56:16.207883 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:56:16.212423 (+ 4540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:16.213601 (+ 1178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:16.213612 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:16.213616 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:16.213835 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:16.214756 (+ 921us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:16.214762 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:16.214766 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:16.215346 (+ 580us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:56:16.215747 (+ 401us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:16.216293 (+ 546us) server_negotiation.cc:299] Negotiation successful
0807 22:56:16.216567 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
W20260807 22:56:16.218058 28045 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45298
I20260807 22:56:16.257602 28638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:16.221787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59240 (local address 127.25.94.62:43863)
0807 22:56:16.237045 (+ 15258us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:16.237064 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:16.237114 (+ 50us) server_negotiation.cc:407] Connection header received
0807 22:56:16.237255 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:16.237263 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:16.237383 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:16.237578 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:56:16.240033 (+ 2455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:16.241361 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:16.243372 (+ 2011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:16.244093 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:16.244239 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:16.247696 (+ 3457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:56:16.247704 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:56:16.247709 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:56:16.247777 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:56:16.253841 (+ 6064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:16.254925 (+ 1084us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:16.254931 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:16.254935 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:16.255135 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:16.255631 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:16.255637 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:16.255641 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:16.256118 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:56:16.256409 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:16.256883 (+ 474us) server_negotiation.cc:299] Negotiation successful
0807 22:56:16.257140 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15085,"thread_start_us":14886,"threads_started":1}
W20260807 22:56:16.258380 27738 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59240
I20260807 22:56:16.262756 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 27840
I20260807 22:56:16.283375 28503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:16.263957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45302 (local address 127.25.94.1:46797)
0807 22:56:16.264191 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:16.264198 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:16.264240 (+ 42us) server_negotiation.cc:407] Connection header received
0807 22:56:16.264462 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:16.264469 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:16.264556 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:16.264629 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:56:16.266526 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:16.267755 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:16.269550 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:16.270410 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:16.270546 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:16.273948 (+ 3402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:56:16.273958 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:56:16.273963 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:56:16.274008 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:56:16.278186 (+ 4178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:16.279190 (+ 1004us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:16.279197 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:16.279201 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:16.279390 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:16.279891 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:16.279897 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:16.279900 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:16.280322 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:56:16.280751 (+ 429us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:16.282391 (+ 1640us) server_negotiation.cc:299] Negotiation successful
0807 22:56:16.282736 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
W20260807 22:56:16.284663 28045 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45302
W20260807 22:56:16.285640 25976 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 27840: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 22:56:16.286067 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 27840
I20260807 22:56:16.321975 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 28063
Aug 07 22:56:16 dist-test-slave-z8x9 krb5kdc[27673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143374, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.25.94.2@KRBTEST.COM
I20260807 22:56:16.339017 28502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:16.322745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45544 (local address 127.25.94.2:44545)
0807 22:56:16.322943 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:16.322948 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:16.323184 (+ 236us) server_negotiation.cc:407] Connection header received
0807 22:56:16.323393 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:16.323399 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:16.323479 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:16.323597 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:56:16.324933 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:16.325865 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:16.327098 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:16.327535 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:16.327674 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:16.332289 (+ 4615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:56:16.332309 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:56:16.332326 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:56:16.332364 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:56:16.335596 (+ 3232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:16.336554 (+ 958us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:16.336563 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:16.336565 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:16.336715 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:16.337327 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:16.337332 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:16.337334 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:16.337820 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:56:16.338083 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:16.338437 (+ 354us) server_negotiation.cc:299] Negotiation successful
0807 22:56:16.338669 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260807 22:56:16.339874 28259 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45544
W20260807 22:56:16.340801 25976 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 28063: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 22:56:16.341018 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 28063
I20260807 22:56:16.371619 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 28277
Aug 07 22:56:16 dist-test-slave-z8x9 krb5kdc[27673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143374, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.25.94.3@KRBTEST.COM
I20260807 22:56:16.390386 28583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:16.372405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56698 (local address 127.25.94.3:38191)
0807 22:56:16.372597 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:16.372602 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:16.372688 (+ 86us) server_negotiation.cc:407] Connection header received
0807 22:56:16.372886 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:16.372890 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:16.372960 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:16.373051 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:56:16.374806 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:16.375648 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:16.377646 (+ 1998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:16.378124 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:16.378229 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:16.383874 (+ 5645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:56:16.383894 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:56:16.383911 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:56:16.383951 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:56:16.387268 (+ 3317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:16.388177 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:16.388185 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:16.388187 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:16.388335 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:16.388766 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:16.388770 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:16.388772 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:16.389308 (+ 536us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:56:16.389607 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:16.389903 (+ 296us) server_negotiation.cc:299] Negotiation successful
0807 22:56:16.390081 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260807 22:56:16.391204 28468 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56698
W20260807 22:56:16.391925 25976 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 28277: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 22:56:16.392081 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 28277
I20260807 22:56:16.417868 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 27691
I20260807 22:56:16.419102 27797 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 22:56:16.537333 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 27691
2026-08-07T22:56:16Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (5427 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786143361779964-25976-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 22:56:16 dist-test-slave-z8x9 krb5kdc[28646](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 22:56:16 dist-test-slave-z8x9 krb5kdc[28646](info): set up 2 sockets
Aug 07 22:56:16 dist-test-slave-z8x9 krb5kdc[28646](info): commencing operation
krb5kdc: starting...
W20260807 22:56:17.868577 25976 mini_kdc.cc:121] Time spent starting KDC: real 1.277s user 0.002s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786143361779964-25976-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786143361779964-25976-0/krb5kdc/test-user.keytab.
Aug 07 22:56:18 dist-test-slave-z8x9 krb5kdc[28646](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143378, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T22:56:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T22:56:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.94.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.62.keytab.
Entry for principal kudu/127.25.94.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.62.keytab.
I20260807 22:56:18.086148 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786143361779964-25976-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786143361779964-25976-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786143361779964-25976-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.62:34227
--webserver_interface=127.25.94.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.62.keytab
--principal=kudu/127.25.94.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.20:46393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.94.62:34227
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786143361779964-25976-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786143361779964-25976-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786143361779964-25976-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:56:18.438077 28675 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:56:18.438465 28675 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:56:18.438644 28675 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:56:18.445971 28675 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 22:56:18.446151 28675 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260807 22:56:18.446249 28675 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:56:18.446345 28675 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:56:18.446449 28675 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 22:56:18.446543 28675 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 22:56:18.456545 28675 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:46393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786143361779964-25976-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786143361779964-25976-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.94.62:34227
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.62.keytab
--principal=kudu/127.25.94.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.62:34227
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.94.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786143361779964-25976-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 22:56:18.457998 28675 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:56:18.459386 28675 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:56:18.472252 28682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:56:18.472234 28683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:56:18.472338 28686 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:56:18.473024 28675 server_base.cc:1034] running on GCE node
I20260807 22:56:18.473887 28675 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:56:18.475303 28675 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:56:18.476555 28675 hybrid_clock.cc:648] HybridClock initialized: now 1786143378476489 us; error 35 us; skew 500 ppm
I20260807 22:56:18.476869 28675 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 22:56:18 dist-test-slave-z8x9 krb5kdc[28646](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143378, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:56:18.480384 28675 init.cc:377] Logged in from keytab as kudu/127.25.94.62@KRBTEST.COM (short username kudu)
I20260807 22:56:18.482039 28675 webserver.cc:460] Webserver started at http://127.25.94.62:41577/ using document root <none> and password file <none>
I20260807 22:56:18.482798 28675 fs_manager.cc:359] Metadata directory not provided
I20260807 22:56:18.482923 28675 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:56:18.483284 28675 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:56:18.486903 28675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786143361779964-25976-0/minicluster-data/master-0/data/instance:
uuid: "1c2a3363ee8640eeb7d932cc0d4c104b"
format_stamp: "Formatted at 2026-08-07 22:56:18 on dist-test-slave-z8x9"
server_key: "89a0e4738134f0c804198b6e9b0cb91e"
server_key_iv: "1e819e5281edc5c9252aa565cc17ccce"
server_key_version: "encryptionkey@0"
I20260807 22:56:18.488108 28675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786143361779964-25976-0/minicluster-data/master-0/wal/instance:
uuid: "1c2a3363ee8640eeb7d932cc0d4c104b"
format_stamp: "Formatted at 2026-08-07 22:56:18 on dist-test-slave-z8x9"
server_key: "89a0e4738134f0c804198b6e9b0cb91e"
server_key_iv: "1e819e5281edc5c9252aa565cc17ccce"
server_key_version: "encryptionkey@0"
I20260807 22:56:18.493854 28675 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260807 22:56:18.497596 28694 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:56:18.499009 28675 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260807 22:56:18.499221 28675 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786143361779964-25976-0/minicluster-data/master-0/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786143361779964-25976-0/minicluster-data/master-0/wal
uuid: "1c2a3363ee8640eeb7d932cc0d4c104b"
format_stamp: "Formatted at 2026-08-07 22:56:18 on dist-test-slave-z8x9"
server_key: "89a0e4738134f0c804198b6e9b0cb91e"
server_key_iv: "1e819e5281edc5c9252aa565cc17ccce"
server_key_version: "encryptionkey@0"
I20260807 22:56:18.499485 28675 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786143361779964-25976-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786143361779964-25976-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786143361779964-25976-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:56:18.512571 28675 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:56:18.517376 28675 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:56:18.517674 28675 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:56:18.540092 28675 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.62:34227
I20260807 22:56:18.540112 28802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.62:34227 every 8 connection(s)
I20260807 22:56:18.541577 28675 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
I20260807 22:56:18.545424 28805 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:56:18.548643 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 28675
I20260807 22:56:18.548892 25976 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786143361779964-25976-0/minicluster-data/master-0/wal/instance
I20260807 22:56:18.549754 25976 external_mini_cluster.cc:1428] Setting key a38ace59ab1edae22e33a144b1269334
I20260807 22:56:18.558399 28805 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c2a3363ee8640eeb7d932cc0d4c104b: Bootstrap starting.
I20260807 22:56:18.561736 28805 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1c2a3363ee8640eeb7d932cc0d4c104b: Neither blocks nor log segments found. Creating new log.
I20260807 22:56:18.562858 28805 log.cc:824] T 00000000000000000000000000000000 P 1c2a3363ee8640eeb7d932cc0d4c104b: Log is configured to *not* fsync() on all Append() calls
Aug 07 22:56:18 dist-test-slave-z8x9 krb5kdc[28646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143378, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.62@KRBTEST.COM
I20260807 22:56:18.566026 28805 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c2a3363ee8640eeb7d932cc0d4c104b: No bootstrap required, opened a new log
I20260807 22:56:18.573844 28810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:18.552197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42572 (local address 127.25.94.62:34227)
0807 22:56:18.553016 (+ 819us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:18.553030 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:18.553072 (+ 42us) server_negotiation.cc:407] Connection header received
0807 22:56:18.554862 (+ 1790us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:18.554885 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:18.555303 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:18.555734 (+ 431us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:56:18.557325 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:18.558695 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:18.559967 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:18.560662 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:18.560878 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:18.565739 (+ 4861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:56:18.565748 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:56:18.565763 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:56:18.565799 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:56:18.569110 (+ 3311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:18.569947 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:18.569963 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:18.569970 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:18.570140 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:18.570550 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:18.570559 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:18.570562 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:18.571141 (+ 579us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:56:18.571488 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:18.571956 (+ 468us) server_negotiation.cc:299] Negotiation successful
0807 22:56:18.572543 (+ 587us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":606,"thread_start_us":376,"threads_started":1}
I20260807 22:56:18.574481 28805 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1c2a3363ee8640eeb7d932cc0d4c104b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c2a3363ee8640eeb7d932cc0d4c104b" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 34227 } }
I20260807 22:56:18.574865 28805 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1c2a3363ee8640eeb7d932cc0d4c104b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:56:18.574975 28805 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1c2a3363ee8640eeb7d932cc0d4c104b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c2a3363ee8640eeb7d932cc0d4c104b, State: Initialized, Role: FOLLOWER
I20260807 22:56:18.575708 28805 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1c2a3363ee8640eeb7d932cc0d4c104b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c2a3363ee8640eeb7d932cc0d4c104b" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 34227 } }
I20260807 22:56:18.575978 28805 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1c2a3363ee8640eeb7d932cc0d4c104b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 22:56:18.576150 28805 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1c2a3363ee8640eeb7d932cc0d4c104b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 22:56:18.576339 28805 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1c2a3363ee8640eeb7d932cc0d4c104b [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:56:18.578800 28805 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1c2a3363ee8640eeb7d932cc0d4c104b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c2a3363ee8640eeb7d932cc0d4c104b" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 34227 } }
I20260807 22:56:18.579356 28805 leader_election.cc:302] T 00000000000000000000000000000000 P 1c2a3363ee8640eeb7d932cc0d4c104b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1c2a3363ee8640eeb7d932cc0d4c104b; no voters:
I20260807 22:56:18.579946 28805 leader_election.cc:288] T 00000000000000000000000000000000 P 1c2a3363ee8640eeb7d932cc0d4c104b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 22:56:18.580099 28814 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1c2a3363ee8640eeb7d932cc0d4c104b [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:56:18.580790 28814 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1c2a3363ee8640eeb7d932cc0d4c104b [term 1 LEADER]: Becoming Leader. State: Replica: 1c2a3363ee8640eeb7d932cc0d4c104b, State: Running, Role: LEADER
I20260807 22:56:18.581547 28814 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1c2a3363ee8640eeb7d932cc0d4c104b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c2a3363ee8640eeb7d932cc0d4c104b" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 34227 } }
I20260807 22:56:18.582712 28805 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1c2a3363ee8640eeb7d932cc0d4c104b [sys.catalog]: configured and running, proceeding with master startup.
I20260807 22:56:18.584484 28818 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c2a3363ee8640eeb7d932cc0d4c104b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1c2a3363ee8640eeb7d932cc0d4c104b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c2a3363ee8640eeb7d932cc0d4c104b" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 34227 } } }
I20260807 22:56:18.584862 28818 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c2a3363ee8640eeb7d932cc0d4c104b [sys.catalog]: This master's current role is: LEADER
I20260807 22:56:18.585287 28817 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c2a3363ee8640eeb7d932cc0d4c104b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c2a3363ee8640eeb7d932cc0d4c104b. Latest consensus state: current_term: 1 leader_uuid: "1c2a3363ee8640eeb7d932cc0d4c104b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c2a3363ee8640eeb7d932cc0d4c104b" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 34227 } } }
I20260807 22:56:18.586151 28817 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c2a3363ee8640eeb7d932cc0d4c104b [sys.catalog]: This master's current role is: LEADER
I20260807 22:56:18.594674 28821 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 22:56:18.601765 28821 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 22:56:18.610924 28821 catalog_manager.cc:1347] Generated new cluster ID: f16ed9fc7105431ea88ae0250e15224a
I20260807 22:56:18.611061 28821 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 22:56:18.624526 28821 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 22:56:18.626000 28821 catalog_manager.cc:1504] Loading token signing keys...
I20260807 22:56:18.633716 28821 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1c2a3363ee8640eeb7d932cc0d4c104b: Generated new TSK 0
I20260807 22:56:18.634842 28821 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.94.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.1.keytab.
Entry for principal kudu/127.25.94.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.1.keytab.
I20260807 22:56:18.701743 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786143361779964-25976-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786143361779964-25976-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786143361779964-25976-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786143361779964-25976-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.1:0
--local_ip_for_outbound_sockets=127.25.94.1
--webserver_interface=127.25.94.1
--webserver_port=0
--tserver_master_addrs=127.25.94.62:34227
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.1.keytab
--principal=kudu/127.25.94.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.20:46393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786143361779964-25976-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786143361779964-25976-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786143361779964-25976-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:56:19.037986 28843 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:56:19.038323 28843 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:56:19.038545 28843 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:56:19.045926 28843 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:56:19.046090 28843 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:56:19.046253 28843 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260807 22:56:19.046348 28843 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260807 22:56:19.046494 28843 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.1
I20260807 22:56:19.056840 28843 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:46393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786143361779964-25976-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786143361779964-25976-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.1.keytab
--principal=kudu/127.25.94.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786143361779964-25976-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.94.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.25.94.62:34227
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.1
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786143361779964-25976-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 22:56:19.058259 28843 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:56:19.059589 28843 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:56:19.073092 28851 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:56:19.073114 28855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:56:19.073108 28850 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:56:19.074251 28843 server_base.cc:1034] running on GCE node
I20260807 22:56:19.074869 28843 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:56:19.075837 28843 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:56:19.077066 28843 hybrid_clock.cc:648] HybridClock initialized: now 1786143379077015 us; error 44 us; skew 500 ppm
I20260807 22:56:19.077450 28843 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 22:56:19 dist-test-slave-z8x9 krb5kdc[28646](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143379, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:56:19.085413 28843 init.cc:377] Logged in from keytab as kudu/127.25.94.1@KRBTEST.COM (short username kudu)
I20260807 22:56:19.087546 28843 webserver.cc:460] Webserver started at http://127.25.94.1:45743/ using document root <none> and password file <none>
I20260807 22:56:19.088528 28843 fs_manager.cc:359] Metadata directory not provided
I20260807 22:56:19.088713 28843 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:56:19.089071 28843 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:56:19.092353 28843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786143361779964-25976-0/minicluster-data/ts-0/data/instance:
uuid: "856a406170f44f378bfd5638cf75ee7a"
format_stamp: "Formatted at 2026-08-07 22:56:19 on dist-test-slave-z8x9"
server_key: "ac755e76b5e31a3eb06fab79df138da1"
server_key_iv: "931659be6a00bb4775e0a8f44d1d44d0"
server_key_version: "encryptionkey@0"
I20260807 22:56:19.093863 28843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786143361779964-25976-0/minicluster-data/ts-0/wal/instance:
uuid: "856a406170f44f378bfd5638cf75ee7a"
format_stamp: "Formatted at 2026-08-07 22:56:19 on dist-test-slave-z8x9"
server_key: "ac755e76b5e31a3eb06fab79df138da1"
server_key_iv: "931659be6a00bb4775e0a8f44d1d44d0"
server_key_version: "encryptionkey@0"
I20260807 22:56:19.099946 28843 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260807 22:56:19.104070 28868 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:56:19.105719 28843 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260807 22:56:19.105877 28843 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786143361779964-25976-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786143361779964-25976-0/minicluster-data/ts-0/wal
uuid: "856a406170f44f378bfd5638cf75ee7a"
format_stamp: "Formatted at 2026-08-07 22:56:19 on dist-test-slave-z8x9"
server_key: "ac755e76b5e31a3eb06fab79df138da1"
server_key_iv: "931659be6a00bb4775e0a8f44d1d44d0"
server_key_version: "encryptionkey@0"
I20260807 22:56:19.106076 28843 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786143361779964-25976-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786143361779964-25976-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786143361779964-25976-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:56:19.145712 28843 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:56:19.150569 28843 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:56:19.150851 28843 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:56:19.153716 28843 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:56:19.154001 28843 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:56:19.154197 28843 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:56:19.154314 28843 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 07 22:56:19 dist-test-slave-z8x9 krb5kdc[28646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143379, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.1@KRBTEST.COM for kudu/127.25.94.62@KRBTEST.COM
I20260807 22:56:19.182041 28882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:19.154739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.1:58247 (local address 127.25.94.62:34227)
0807 22:56:19.155282 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:19.155289 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:19.156607 (+ 1318us) server_negotiation.cc:407] Connection header received
0807 22:56:19.159922 (+ 3315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:19.159964 (+ 42us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:19.160114 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:19.160303 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:56:19.163402 (+ 3099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:19.164484 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:19.168288 (+ 3804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:19.168714 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:19.168844 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:19.174921 (+ 6077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:56:19.174927 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:56:19.174931 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:56:19.174969 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:56:19.178128 (+ 3159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:19.179295 (+ 1167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:19.179304 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:19.179308 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:19.179504 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:19.180056 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:19.180061 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:19.180065 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:19.180370 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:56:19.180541 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:19.181375 (+ 834us) server_negotiation.cc:299] Negotiation successful
0807 22:56:19.181624 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":265,"threads_started":1}
I20260807 22:56:19.183230 28886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:19.155389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:34227 (local address 127.25.94.1:58247)
0807 22:56:19.156352 (+ 963us) negotiation.cc:107] Waiting for socket to connect
0807 22:56:19.156419 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0807 22:56:19.159415 (+ 2996us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:56:19.160697 (+ 1282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:56:19.160710 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:56:19.161879 (+ 1169us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:56:19.163078 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:19.163121 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:19.164772 (+ 1651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:19.164782 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:19.167988 (+ 3206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:19.168025 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:19.169058 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:19.169070 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:19.169550 (+ 480us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:19.170727 (+ 1177us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:56:19.170787 (+ 60us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:56:19.174594 (+ 3807us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:56:19.178303 (+ 3709us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:56:19.178314 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:56:19.178333 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:56:19.179083 (+ 750us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:56:19.179676 (+ 593us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:56:19.179680 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:56:19.179683 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:56:19.179917 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:56:19.180525 (+ 608us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:56:19.180534 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:56:19.180935 (+ 401us) client_negotiation.cc:769] Sending connection context
0807 22:56:19.181251 (+ 316us) client_negotiation.cc:240] Negotiation successful
0807 22:56:19.181885 (+ 634us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":624,"thread_start_us":362,"threads_started":1}
I20260807 22:56:19.210155 28882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:19.190947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.1:48061 (local address 127.25.94.62:34227)
0807 22:56:19.191157 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:19.191166 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:19.191893 (+ 727us) server_negotiation.cc:407] Connection header received
0807 22:56:19.192106 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:19.192110 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:19.192177 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:19.192267 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:56:19.194066 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:19.195127 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:19.197890 (+ 2763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:19.198752 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:19.198926 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:19.200999 (+ 2073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:56:19.201009 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:56:19.201017 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:56:19.201069 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:56:19.204696 (+ 3627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:19.205715 (+ 1019us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:19.205722 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:19.205728 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:19.205928 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:19.206409 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:19.206416 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:19.206420 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:19.206857 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:56:19.207304 (+ 447us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:19.209335 (+ 2031us) server_negotiation.cc:299] Negotiation successful
0807 22:56:19.209670 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260807 22:56:19.210789 28978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:19.190957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:34227 (local address 127.25.94.1:48061)
0807 22:56:19.191732 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0807 22:56:19.191760 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0807 22:56:19.191971 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:56:19.192340 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:56:19.192348 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:56:19.193196 (+ 848us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:56:19.193760 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:19.193780 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:19.195252 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:19.195260 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:19.197629 (+ 2369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:19.197652 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:19.199010 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:19.199016 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:19.199141 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:19.199935 (+ 794us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:56:19.199954 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:56:19.200740 (+ 786us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:56:19.204859 (+ 4119us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:56:19.204866 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:56:19.204870 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:56:19.205562 (+ 692us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:56:19.206095 (+ 533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:56:19.206103 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:56:19.206109 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:56:19.206278 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:56:19.207251 (+ 973us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:56:19.207258 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:56:19.207597 (+ 339us) client_negotiation.cc:769] Sending connection context
0807 22:56:19.207882 (+ 285us) client_negotiation.cc:240] Negotiation successful
0807 22:56:19.208422 (+ 540us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":392,"threads_started":1}
I20260807 22:56:19.213456 28843 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.1:44541
I20260807 22:56:19.213507 29058 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.1:44541 every 8 connection(s)
I20260807 22:56:19.215013 28843 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786143361779964-25976-0/minicluster-data/ts-0/data/info.pb
I20260807 22:56:19.217502 29059 heartbeater.cc:347] Connected to a master server at 127.25.94.62:34227
I20260807 22:56:19.217751 29059 heartbeater.cc:464] Registering TS with master...
I20260807 22:56:19.218472 29059 heartbeater.cc:511] Master 127.25.94.62:34227 requested a full tablet report, sending...
I20260807 22:56:19.220676 28728 ts_manager.cc:194] Registered new tserver with Master: 856a406170f44f378bfd5638cf75ee7a (127.25.94.1:44541)
I20260807 22:56:19.222306 28728 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.94.1@KRBTEST.COM'} at 127.25.94.1:58247
I20260807 22:56:19.225411 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 28843
I20260807 22:56:19.225600 25976 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786143361779964-25976-0/minicluster-data/ts-0/wal/instance
I20260807 22:56:19.226209 25976 external_mini_cluster.cc:1428] Setting key 865f745c9fc930149a458153f539a78b
WARNING: no policy specified for kudu/127.25.94.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.2.keytab.
Entry for principal kudu/127.25.94.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.2.keytab.
I20260807 22:56:19.273851 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786143361779964-25976-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786143361779964-25976-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786143361779964-25976-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786143361779964-25976-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.2:0
--local_ip_for_outbound_sockets=127.25.94.2
--webserver_interface=127.25.94.2
--webserver_port=0
--tserver_master_addrs=127.25.94.62:34227
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.2.keytab
--principal=kudu/127.25.94.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.20:46393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786143361779964-25976-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786143361779964-25976-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786143361779964-25976-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:56:19.612591 29064 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:56:19.612886 29064 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:56:19.613061 29064 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:56:19.620393 29064 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:56:19.620519 29064 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:56:19.620669 29064 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260807 22:56:19.620999 29064 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260807 22:56:19.621236 29064 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.2
I20260807 22:56:19.631330 29064 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:46393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786143361779964-25976-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786143361779964-25976-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.2.keytab
--principal=kudu/127.25.94.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786143361779964-25976-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.94.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.25.94.62:34227
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.2
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786143361779964-25976-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 22:56:19.632653 29064 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:56:19.634198 29064 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:56:19.647675 29083 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:56:19.647738 29073 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:56:19.647753 29075 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:56:19.648459 29064 server_base.cc:1034] running on GCE node
I20260807 22:56:19.649112 29064 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:56:19.650141 29064 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:56:19.651377 29064 hybrid_clock.cc:648] HybridClock initialized: now 1786143379651337 us; error 55 us; skew 500 ppm
I20260807 22:56:19.651701 29064 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 22:56:19 dist-test-slave-z8x9 krb5kdc[28646](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143379, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:56:19.655061 29064 init.cc:377] Logged in from keytab as kudu/127.25.94.2@KRBTEST.COM (short username kudu)
I20260807 22:56:19.656553 29064 webserver.cc:460] Webserver started at http://127.25.94.2:40593/ using document root <none> and password file <none>
I20260807 22:56:19.657357 29064 fs_manager.cc:359] Metadata directory not provided
I20260807 22:56:19.657519 29064 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:56:19.657846 29064 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:56:19.660776 29064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786143361779964-25976-0/minicluster-data/ts-1/data/instance:
uuid: "9e5fefbbe4064049919e70cf86eb9757"
format_stamp: "Formatted at 2026-08-07 22:56:19 on dist-test-slave-z8x9"
server_key: "1518e150c513c593d366a862cea451de"
server_key_iv: "dee3fcab4527059eec31ad5abf7ec59f"
server_key_version: "encryptionkey@0"
I20260807 22:56:19.661584 29064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786143361779964-25976-0/minicluster-data/ts-1/wal/instance:
uuid: "9e5fefbbe4064049919e70cf86eb9757"
format_stamp: "Formatted at 2026-08-07 22:56:19 on dist-test-slave-z8x9"
server_key: "1518e150c513c593d366a862cea451de"
server_key_iv: "dee3fcab4527059eec31ad5abf7ec59f"
server_key_version: "encryptionkey@0"
I20260807 22:56:19.666640 29064 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260807 22:56:19.670212 29091 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:56:19.671463 29064 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260807 22:56:19.671640 29064 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786143361779964-25976-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786143361779964-25976-0/minicluster-data/ts-1/wal
uuid: "9e5fefbbe4064049919e70cf86eb9757"
format_stamp: "Formatted at 2026-08-07 22:56:19 on dist-test-slave-z8x9"
server_key: "1518e150c513c593d366a862cea451de"
server_key_iv: "dee3fcab4527059eec31ad5abf7ec59f"
server_key_version: "encryptionkey@0"
I20260807 22:56:19.671841 29064 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786143361779964-25976-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786143361779964-25976-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786143361779964-25976-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:56:19.695099 29064 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:56:19.699640 29064 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:56:19.699977 29064 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:56:19.702919 29064 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:56:19.703053 29064 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:56:19.703193 29064 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:56:19.703346 29064 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 07 22:56:19 dist-test-slave-z8x9 krb5kdc[28646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143379, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.2@KRBTEST.COM for kudu/127.25.94.62@KRBTEST.COM
I20260807 22:56:19.728377 28882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:19.703535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.2:44069 (local address 127.25.94.62:34227)
0807 22:56:19.703753 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:19.703760 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:19.705462 (+ 1702us) server_negotiation.cc:407] Connection header received
0807 22:56:19.707641 (+ 2179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:19.707647 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:19.707746 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:19.707860 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:56:19.710522 (+ 2662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:19.711536 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:19.714334 (+ 2798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:19.714995 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:19.715160 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:19.720623 (+ 5463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:56:19.720632 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:56:19.720639 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:56:19.720693 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:56:19.724155 (+ 3462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:19.725490 (+ 1335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:19.725499 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:19.725504 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:19.725668 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:19.726325 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:19.726331 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:19.726334 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:19.726793 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:56:19.726981 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:19.727773 (+ 792us) server_negotiation.cc:299] Negotiation successful
0807 22:56:19.728013 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260807 22:56:19.729729 29103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:19.704225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:34227 (local address 127.25.94.2:44069)
0807 22:56:19.705294 (+ 1069us) negotiation.cc:107] Waiting for socket to connect
0807 22:56:19.705344 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0807 22:56:19.707326 (+ 1982us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:56:19.708169 (+ 843us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:56:19.708184 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:56:19.709077 (+ 893us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:56:19.710263 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:19.710297 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:19.711706 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:19.711712 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:19.714133 (+ 2421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:19.714150 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:19.715337 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:19.715348 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:19.715574 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:19.716521 (+ 947us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:56:19.716551 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:56:19.720338 (+ 3787us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:56:19.724329 (+ 3991us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:56:19.724339 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:56:19.724361 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:56:19.725102 (+ 741us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:56:19.725808 (+ 706us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:56:19.725813 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:56:19.725821 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:56:19.726178 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:56:19.726948 (+ 770us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:56:19.726957 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:56:19.727373 (+ 416us) client_negotiation.cc:769] Sending connection context
0807 22:56:19.727656 (+ 283us) client_negotiation.cc:240] Negotiation successful
0807 22:56:19.728214 (+ 558us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":752,"thread_start_us":493,"threads_started":1}
I20260807 22:56:19.751791 28882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:19.737304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.2:45547 (local address 127.25.94.62:34227)
0807 22:56:19.737482 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:19.737490 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:19.737591 (+ 101us) server_negotiation.cc:407] Connection header received
0807 22:56:19.737900 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:19.737906 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:19.737997 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:19.738098 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:56:19.739836 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:19.740891 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:19.742796 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:19.743200 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:19.743332 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:19.745202 (+ 1870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:56:19.745209 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:56:19.745215 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:56:19.745258 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:56:19.748704 (+ 3446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:19.749689 (+ 985us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:19.749696 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:19.749700 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:19.749845 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:19.750241 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:19.750246 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:19.750249 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:19.750627 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:56:19.750787 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:19.751201 (+ 414us) server_negotiation.cc:299] Negotiation successful
0807 22:56:19.751437 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260807 22:56:19.751925 29209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:19.736632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:34227 (local address 127.25.94.2:45547)
0807 22:56:19.737481 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0807 22:56:19.737504 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0807 22:56:19.737709 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:56:19.738279 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:56:19.738285 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:56:19.739123 (+ 838us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:56:19.739619 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:19.739639 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:19.741034 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:19.741039 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:19.742646 (+ 1607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:19.742658 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:19.743405 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:19.743412 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:19.743575 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:19.744227 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:56:19.744246 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:56:19.744967 (+ 721us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:56:19.748893 (+ 3926us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:56:19.748897 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:56:19.748900 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:56:19.749568 (+ 668us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:56:19.749965 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:56:19.749970 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:56:19.749973 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:56:19.750109 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:56:19.750758 (+ 649us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:56:19.750763 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:56:19.750946 (+ 183us) client_negotiation.cc:769] Sending connection context
0807 22:56:19.751090 (+ 144us) client_negotiation.cc:240] Negotiation successful
0807 22:56:19.751439 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":667,"thread_start_us":848,"threads_started":1}
I20260807 22:56:19.757575 29064 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.2:45043
I20260807 22:56:19.757625 29264 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.2:45043 every 8 connection(s)
I20260807 22:56:19.758901 29064 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786143361779964-25976-0/minicluster-data/ts-1/data/info.pb
I20260807 22:56:19.760905 29265 heartbeater.cc:347] Connected to a master server at 127.25.94.62:34227
I20260807 22:56:19.761163 29265 heartbeater.cc:464] Registering TS with master...
I20260807 22:56:19.761854 29265 heartbeater.cc:511] Master 127.25.94.62:34227 requested a full tablet report, sending...
I20260807 22:56:19.763231 28728 ts_manager.cc:194] Registered new tserver with Master: 9e5fefbbe4064049919e70cf86eb9757 (127.25.94.2:45043)
I20260807 22:56:19.764035 28728 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.94.2@KRBTEST.COM'} at 127.25.94.2:44069
I20260807 22:56:19.766330 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 29064
I20260807 22:56:19.766515 25976 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786143361779964-25976-0/minicluster-data/ts-1/wal/instance
I20260807 22:56:19.766954 25976 external_mini_cluster.cc:1428] Setting key 3f32cb7aef39efb9f94c8248e48e7bf4
WARNING: no policy specified for kudu/127.25.94.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.3.keytab.
Entry for principal kudu/127.25.94.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.3.keytab.
I20260807 22:56:19.813275 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786143361779964-25976-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786143361779964-25976-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786143361779964-25976-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786143361779964-25976-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.3:0
--local_ip_for_outbound_sockets=127.25.94.3
--webserver_interface=127.25.94.3
--webserver_port=0
--tserver_master_addrs=127.25.94.62:34227
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.3.keytab
--principal=kudu/127.25.94.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.20:46393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786143361779964-25976-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786143361779964-25976-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786143361779964-25976-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:56:20.158046 29269 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:56:20.158409 29269 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:56:20.158595 29269 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:56:20.167241 29269 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:56:20.167397 29269 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:56:20.167577 29269 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260807 22:56:20.167642 29269 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260807 22:56:20.167769 29269 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.3
I20260807 22:56:20.179124 29269 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:46393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786143361779964-25976-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786143361779964-25976-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.3.keytab
--principal=kudu/127.25.94.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786143361779964-25976-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.94.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.25.94.62:34227
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.3
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786143361779964-25976-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 22:56:20.180655 29269 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:56:20.182184 29269 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:56:20.198405 29275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:56:20.198352 29278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:56:20.198423 29276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:56:20.199434 29269 server_base.cc:1034] running on GCE node
I20260807 22:56:20.200258 29269 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:56:20.201445 29269 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:56:20.202704 29269 hybrid_clock.cc:648] HybridClock initialized: now 1786143380202651 us; error 40 us; skew 500 ppm
I20260807 22:56:20.203146 29269 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 22:56:20 dist-test-slave-z8x9 krb5kdc[28646](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143380, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:56:20.206715 29269 init.cc:377] Logged in from keytab as kudu/127.25.94.3@KRBTEST.COM (short username kudu)
I20260807 22:56:20.208308 29269 webserver.cc:460] Webserver started at http://127.25.94.3:34183/ using document root <none> and password file <none>
I20260807 22:56:20.209015 29269 fs_manager.cc:359] Metadata directory not provided
I20260807 22:56:20.209151 29269 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:56:20.209575 29269 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:56:20.213115 29269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786143361779964-25976-0/minicluster-data/ts-2/data/instance:
uuid: "bb456655a12a41039f76160cfda886ea"
format_stamp: "Formatted at 2026-08-07 22:56:20 on dist-test-slave-z8x9"
server_key: "a84071f03fc14ed60f7555aaa11da485"
server_key_iv: "bc1d0f48dda87e0d32444f697fb9a33c"
server_key_version: "encryptionkey@0"
I20260807 22:56:20.214319 29269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786143361779964-25976-0/minicluster-data/ts-2/wal/instance:
uuid: "bb456655a12a41039f76160cfda886ea"
format_stamp: "Formatted at 2026-08-07 22:56:20 on dist-test-slave-z8x9"
server_key: "a84071f03fc14ed60f7555aaa11da485"
server_key_iv: "bc1d0f48dda87e0d32444f697fb9a33c"
server_key_version: "encryptionkey@0"
I20260807 22:56:20.220842 29269 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260807 22:56:20.225301 29059 heartbeater.cc:503] Master 127.25.94.62:34227 was elected leader, sending a full tablet report...
I20260807 22:56:20.225756 29292 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:56:20.227633 29269 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260807 22:56:20.227845 29269 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786143361779964-25976-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786143361779964-25976-0/minicluster-data/ts-2/wal
uuid: "bb456655a12a41039f76160cfda886ea"
format_stamp: "Formatted at 2026-08-07 22:56:20 on dist-test-slave-z8x9"
server_key: "a84071f03fc14ed60f7555aaa11da485"
server_key_iv: "bc1d0f48dda87e0d32444f697fb9a33c"
server_key_version: "encryptionkey@0"
I20260807 22:56:20.228077 29269 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786143361779964-25976-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786143361779964-25976-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786143361779964-25976-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:56:20.268922 29269 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:56:20.274339 29269 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:56:20.274682 29269 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:56:20.278076 29269 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:56:20.278220 29269 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:56:20.278419 29269 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:56:20.278527 29269 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 07 22:56:20 dist-test-slave-z8x9 krb5kdc[28646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143380, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.3@KRBTEST.COM for kudu/127.25.94.62@KRBTEST.COM
I20260807 22:56:20.303840 29301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:20.278450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.3:49423 (local address 127.25.94.62:34227)
0807 22:56:20.279005 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:20.279014 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:20.280184 (+ 1170us) server_negotiation.cc:407] Connection header received
0807 22:56:20.282317 (+ 2133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:20.282323 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:20.282422 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:20.282545 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:56:20.284912 (+ 2367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:20.286056 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:20.288879 (+ 2823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:20.289585 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:20.289736 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:20.294687 (+ 4951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:56:20.294692 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:56:20.294696 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:56:20.294734 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:56:20.299288 (+ 4554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:20.300623 (+ 1335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:20.300631 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:20.300635 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:20.300878 (+ 243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:20.301655 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:20.301661 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:20.301664 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:20.301993 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:56:20.302195 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:20.303099 (+ 904us) server_negotiation.cc:299] Negotiation successful
0807 22:56:20.303401 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":269,"threads_started":1}
I20260807 22:56:20.304996 29303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:20.279211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:34227 (local address 127.25.94.3:49423)
0807 22:56:20.280018 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0807 22:56:20.280057 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0807 22:56:20.282012 (+ 1955us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:56:20.282811 (+ 799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:56:20.282827 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:56:20.283740 (+ 913us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:56:20.284703 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:20.284727 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:20.286273 (+ 1546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:20.286278 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:20.288615 (+ 2337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:20.288641 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:20.289855 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:20.289864 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:20.290200 (+ 336us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:20.291165 (+ 965us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:56:20.291211 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:56:20.294388 (+ 3177us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:56:20.299592 (+ 5204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:56:20.299608 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:56:20.299636 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:56:20.300452 (+ 816us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:56:20.301066 (+ 614us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:56:20.301075 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:56:20.301082 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:56:20.301494 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:56:20.302173 (+ 679us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:56:20.302182 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:56:20.302591 (+ 409us) client_negotiation.cc:769] Sending connection context
0807 22:56:20.302956 (+ 365us) client_negotiation.cc:240] Negotiation successful
0807 22:56:20.303577 (+ 621us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":367,"threads_started":1}
I20260807 22:56:20.330264 29301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:20.312561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.3:60351 (local address 127.25.94.62:34227)
0807 22:56:20.312794 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:20.312802 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:20.313661 (+ 859us) server_negotiation.cc:407] Connection header received
0807 22:56:20.313913 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:20.313920 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:20.314023 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:20.314143 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:56:20.315802 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:20.316916 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:20.319253 (+ 2337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:20.319908 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:20.320062 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:20.322403 (+ 2341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:56:20.322412 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:56:20.322421 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:56:20.322477 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:56:20.326731 (+ 4254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:20.327857 (+ 1126us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:20.327863 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:20.327869 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:20.328043 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:20.328461 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:20.328469 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:20.328474 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:20.328907 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:56:20.329092 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:20.329647 (+ 555us) server_negotiation.cc:299] Negotiation successful
0807 22:56:20.329893 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260807 22:56:20.330365 29380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:20.312608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:34227 (local address 127.25.94.3:60351)
0807 22:56:20.313526 (+ 918us) negotiation.cc:107] Waiting for socket to connect
0807 22:56:20.313544 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 22:56:20.313747 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:56:20.314217 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:56:20.314225 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:56:20.315043 (+ 818us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:56:20.315544 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:20.315562 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:20.317129 (+ 1567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:20.317135 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:20.319027 (+ 1892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:20.319047 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:20.320128 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:20.320138 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:20.320360 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:20.321076 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:56:20.321095 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:56:20.322162 (+ 1067us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:56:20.327002 (+ 4840us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:56:20.327014 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:56:20.327020 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:56:20.327686 (+ 666us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:56:20.328164 (+ 478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:56:20.328168 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:56:20.328170 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:56:20.328328 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:56:20.329053 (+ 725us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:56:20.329057 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:56:20.329340 (+ 283us) client_negotiation.cc:769] Sending connection context
0807 22:56:20.329525 (+ 185us) client_negotiation.cc:240] Negotiation successful
0807 22:56:20.329885 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":769,"thread_start_us":539,"threads_started":1}
I20260807 22:56:20.342774 29269 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.3:45619
I20260807 22:56:20.342830 29473 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.3:45619 every 8 connection(s)
I20260807 22:56:20.344257 29269 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786143361779964-25976-0/minicluster-data/ts-2/data/info.pb
I20260807 22:56:20.346488 29474 heartbeater.cc:347] Connected to a master server at 127.25.94.62:34227
I20260807 22:56:20.346725 29474 heartbeater.cc:464] Registering TS with master...
I20260807 22:56:20.346953 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 29269
I20260807 22:56:20.347136 25976 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786143361779964-25976-0/minicluster-data/ts-2/wal/instance
I20260807 22:56:20.347484 29474 heartbeater.cc:511] Master 127.25.94.62:34227 requested a full tablet report, sending...
I20260807 22:56:20.347663 25976 external_mini_cluster.cc:1428] Setting key 826a5bda15eb64fc255f7f808b378eaf
I20260807 22:56:20.349205 28727 ts_manager.cc:194] Registered new tserver with Master: bb456655a12a41039f76160cfda886ea (127.25.94.3:45619)
I20260807 22:56:20.350318 28727 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.94.3@KRBTEST.COM'} at 127.25.94.3:49423
I20260807 22:56:20.351219 25976 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 07 22:56:20 dist-test-slave-z8x9 krb5kdc[28646](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143380, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 07 22:56:20 dist-test-slave-z8x9 krb5kdc[28646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143380, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.62@KRBTEST.COM
I20260807 22:56:20.402281 29301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:20.388061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42576 (local address 127.25.94.62:34227)
0807 22:56:20.388328 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:20.388334 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:20.388517 (+ 183us) server_negotiation.cc:407] Connection header received
0807 22:56:20.388669 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:20.388685 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:20.388768 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:20.388859 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:56:20.390414 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:20.391418 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:20.392736 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:20.393132 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:20.393305 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:20.397225 (+ 3920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:56:20.397231 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:56:20.397235 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:56:20.397270 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:56:20.399833 (+ 2563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:20.400469 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:20.400472 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:20.400474 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:20.400610 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:20.400936 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:20.400939 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:20.400941 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:20.401282 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:56:20.401494 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:20.401721 (+ 227us) server_negotiation.cc:299] Negotiation successful
0807 22:56:20.401898 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260807 22:56:20.407019 28727 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42576:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 22:56:20.414237 28727 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 22:56:20.438818 29501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:20.429325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59690 (local address 127.25.94.3:45619)
0807 22:56:20.431862 (+ 2537us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:20.431870 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:20.431903 (+ 33us) server_negotiation.cc:407] Connection header received
0807 22:56:20.431994 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:20.432001 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:20.432202 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:20.432386 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:56:20.433063 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:20.434508 (+ 1445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:20.436546 (+ 2038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:20.437590 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:20.437738 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:20.438189 (+ 451us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:20.438263 (+ 74us) server_negotiation.cc:299] Negotiation successful
0807 22:56:20.438445 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2353,"thread_start_us":2141,"threads_started":1}
I20260807 22:56:20.438881 29492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:20.426988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.2:45043 (local address 127.0.0.1:55172)
0807 22:56:20.428028 (+ 1040us) negotiation.cc:107] Waiting for socket to connect
0807 22:56:20.428068 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0807 22:56:20.428231 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:56:20.430666 (+ 2435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:56:20.430673 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:56:20.430705 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:56:20.431153 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:20.431169 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:20.433087 (+ 1918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:20.433092 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:20.436017 (+ 2925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:20.436034 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:20.437949 (+ 1915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:20.437956 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:20.438156 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:20.438193 (+ 37us) client_negotiation.cc:769] Sending connection context
0807 22:56:20.438259 (+ 66us) client_negotiation.cc:240] Negotiation successful
0807 22:56:20.438463 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":785,"mutex_wait_us":144,"thread_start_us":873,"threads_started":1}
I20260807 22:56:20.439435 29498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:20.428368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51654 (local address 127.25.94.1:44541)
0807 22:56:20.429156 (+ 788us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:20.429193 (+ 37us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:20.430018 (+ 825us) server_negotiation.cc:407] Connection header received
0807 22:56:20.430122 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:20.430133 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:20.430383 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:20.430585 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:56:20.431562 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:20.433029 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:20.436591 (+ 3562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:20.438149 (+ 1558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:20.438278 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:20.438485 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:20.438820 (+ 335us) server_negotiation.cc:299] Negotiation successful
0807 22:56:20.439105 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":603,"thread_start_us":473,"threads_started":1}
I20260807 22:56:20.439471 29499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:20.428633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.3:45619 (local address 127.0.0.1:59690)
0807 22:56:20.429970 (+ 1337us) negotiation.cc:107] Waiting for socket to connect
0807 22:56:20.429988 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 22:56:20.430083 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:56:20.432395 (+ 2312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:56:20.432400 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:56:20.432429 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:56:20.432811 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:20.432823 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:20.434673 (+ 1850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:20.434678 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:20.436395 (+ 1717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:20.436407 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:20.437793 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:20.437800 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:20.438017 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:20.438077 (+ 60us) client_negotiation.cc:769] Sending connection context
0807 22:56:20.438214 (+ 137us) client_negotiation.cc:240] Negotiation successful
0807 22:56:20.439105 (+ 891us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1218,"mutex_wait_us":584,"thread_start_us":1082,"threads_started":1}
I20260807 22:56:20.439999 29495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:20.427635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.1:44541 (local address 127.0.0.1:51654)
0807 22:56:20.429702 (+ 2067us) negotiation.cc:107] Waiting for socket to connect
0807 22:56:20.429724 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 22:56:20.429878 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:56:20.430825 (+ 947us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:56:20.430829 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:56:20.430855 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:56:20.431359 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:20.431372 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:20.433191 (+ 1819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:20.433195 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:20.436410 (+ 3215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:20.436423 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:20.438456 (+ 2033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:20.438462 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:20.438626 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:20.438655 (+ 29us) client_negotiation.cc:769] Sending connection context
0807 22:56:20.438756 (+ 101us) client_negotiation.cc:240] Negotiation successful
0807 22:56:20.439569 (+ 813us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1891,"mutex_wait_us":533,"thread_start_us":1728,"threads_started":1}
I20260807 22:56:20.441074 29494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:20.427219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55172 (local address 127.25.94.2:45043)
0807 22:56:20.428818 (+ 1599us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:20.428827 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:20.428856 (+ 29us) server_negotiation.cc:407] Connection header received
0807 22:56:20.430215 (+ 1359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:20.430224 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:20.430433 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:20.431377 (+ 944us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:56:20.431620 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:20.432973 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:20.436199 (+ 3226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:20.437681 (+ 1482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:20.440354 (+ 2673us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:20.440550 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:20.440621 (+ 71us) server_negotiation.cc:299] Negotiation successful
0807 22:56:20.440775 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1415,"thread_start_us":1280,"threads_started":1}
I20260807 22:56:20.443869 29163 tablet_service.cc:1467] Processing CreateTablet for tablet 24775f48f02549dab3e902d0663dc6a2 (DEFAULT_TABLE table=test-table [id=74072c6c61cf421abf36318aee46878a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:56:20.443903 29362 tablet_service.cc:1467] Processing CreateTablet for tablet 24775f48f02549dab3e902d0663dc6a2 (DEFAULT_TABLE table=test-table [id=74072c6c61cf421abf36318aee46878a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:56:20.443919 28950 tablet_service.cc:1467] Processing CreateTablet for tablet 24775f48f02549dab3e902d0663dc6a2 (DEFAULT_TABLE table=test-table [id=74072c6c61cf421abf36318aee46878a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:56:20.445320 29163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24775f48f02549dab3e902d0663dc6a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:56:20.445317 29362 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24775f48f02549dab3e902d0663dc6a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:56:20.445493 28950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24775f48f02549dab3e902d0663dc6a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:56:20.456017 29502 tablet_bootstrap.cc:492] T 24775f48f02549dab3e902d0663dc6a2 P 9e5fefbbe4064049919e70cf86eb9757: Bootstrap starting.
I20260807 22:56:20.459959 29502 tablet_bootstrap.cc:654] T 24775f48f02549dab3e902d0663dc6a2 P 9e5fefbbe4064049919e70cf86eb9757: Neither blocks nor log segments found. Creating new log.
I20260807 22:56:20.460031 29504 tablet_bootstrap.cc:492] T 24775f48f02549dab3e902d0663dc6a2 P bb456655a12a41039f76160cfda886ea: Bootstrap starting.
I20260807 22:56:20.460754 29503 tablet_bootstrap.cc:492] T 24775f48f02549dab3e902d0663dc6a2 P 856a406170f44f378bfd5638cf75ee7a: Bootstrap starting.
I20260807 22:56:20.461514 29502 log.cc:824] T 24775f48f02549dab3e902d0663dc6a2 P 9e5fefbbe4064049919e70cf86eb9757: Log is configured to *not* fsync() on all Append() calls
I20260807 22:56:20.464290 29504 tablet_bootstrap.cc:654] T 24775f48f02549dab3e902d0663dc6a2 P bb456655a12a41039f76160cfda886ea: Neither blocks nor log segments found. Creating new log.
I20260807 22:56:20.464464 29503 tablet_bootstrap.cc:654] T 24775f48f02549dab3e902d0663dc6a2 P 856a406170f44f378bfd5638cf75ee7a: Neither blocks nor log segments found. Creating new log.
I20260807 22:56:20.464911 29502 tablet_bootstrap.cc:492] T 24775f48f02549dab3e902d0663dc6a2 P 9e5fefbbe4064049919e70cf86eb9757: No bootstrap required, opened a new log
I20260807 22:56:20.465220 29502 ts_tablet_manager.cc:1397] T 24775f48f02549dab3e902d0663dc6a2 P 9e5fefbbe4064049919e70cf86eb9757: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260807 22:56:20.465554 29504 log.cc:824] T 24775f48f02549dab3e902d0663dc6a2 P bb456655a12a41039f76160cfda886ea: Log is configured to *not* fsync() on all Append() calls
I20260807 22:56:20.465721 29503 log.cc:824] T 24775f48f02549dab3e902d0663dc6a2 P 856a406170f44f378bfd5638cf75ee7a: Log is configured to *not* fsync() on all Append() calls
I20260807 22:56:20.468921 29504 tablet_bootstrap.cc:492] T 24775f48f02549dab3e902d0663dc6a2 P bb456655a12a41039f76160cfda886ea: No bootstrap required, opened a new log
I20260807 22:56:20.469269 29504 ts_tablet_manager.cc:1397] T 24775f48f02549dab3e902d0663dc6a2 P bb456655a12a41039f76160cfda886ea: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260807 22:56:20.476747 29503 tablet_bootstrap.cc:492] T 24775f48f02549dab3e902d0663dc6a2 P 856a406170f44f378bfd5638cf75ee7a: No bootstrap required, opened a new log
I20260807 22:56:20.476461 29502 raft_consensus.cc:348] T 24775f48f02549dab3e902d0663dc6a2 P 9e5fefbbe4064049919e70cf86eb9757 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e5fefbbe4064049919e70cf86eb9757" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 45043 } } peers { permanent_uuid: "856a406170f44f378bfd5638cf75ee7a" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44541 } } peers { permanent_uuid: "bb456655a12a41039f76160cfda886ea" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 45619 } }
I20260807 22:56:20.477059 29502 raft_consensus.cc:374] T 24775f48f02549dab3e902d0663dc6a2 P 9e5fefbbe4064049919e70cf86eb9757 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:56:20.477128 29503 ts_tablet_manager.cc:1397] T 24775f48f02549dab3e902d0663dc6a2 P 856a406170f44f378bfd5638cf75ee7a: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.000s
I20260807 22:56:20.477239 29502 raft_consensus.cc:729] T 24775f48f02549dab3e902d0663dc6a2 P 9e5fefbbe4064049919e70cf86eb9757 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e5fefbbe4064049919e70cf86eb9757, State: Initialized, Role: FOLLOWER
I20260807 22:56:20.477967 29502 consensus_queue.cc:260] T 24775f48f02549dab3e902d0663dc6a2 P 9e5fefbbe4064049919e70cf86eb9757 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e5fefbbe4064049919e70cf86eb9757" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 45043 } } peers { permanent_uuid: "856a406170f44f378bfd5638cf75ee7a" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44541 } } peers { permanent_uuid: "bb456655a12a41039f76160cfda886ea" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 45619 } }
I20260807 22:56:20.479226 29265 heartbeater.cc:503] Master 127.25.94.62:34227 was elected leader, sending a full tablet report...
I20260807 22:56:20.479805 29504 raft_consensus.cc:348] T 24775f48f02549dab3e902d0663dc6a2 P bb456655a12a41039f76160cfda886ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e5fefbbe4064049919e70cf86eb9757" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 45043 } } peers { permanent_uuid: "856a406170f44f378bfd5638cf75ee7a" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44541 } } peers { permanent_uuid: "bb456655a12a41039f76160cfda886ea" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 45619 } }
I20260807 22:56:20.480249 29504 raft_consensus.cc:374] T 24775f48f02549dab3e902d0663dc6a2 P bb456655a12a41039f76160cfda886ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:56:20.480373 29504 raft_consensus.cc:729] T 24775f48f02549dab3e902d0663dc6a2 P bb456655a12a41039f76160cfda886ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb456655a12a41039f76160cfda886ea, State: Initialized, Role: FOLLOWER
I20260807 22:56:20.481087 29504 consensus_queue.cc:260] T 24775f48f02549dab3e902d0663dc6a2 P bb456655a12a41039f76160cfda886ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e5fefbbe4064049919e70cf86eb9757" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 45043 } } peers { permanent_uuid: "856a406170f44f378bfd5638cf75ee7a" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44541 } } peers { permanent_uuid: "bb456655a12a41039f76160cfda886ea" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 45619 } }
I20260807 22:56:20.481611 29502 ts_tablet_manager.cc:1428] T 24775f48f02549dab3e902d0663dc6a2 P 9e5fefbbe4064049919e70cf86eb9757: Time spent starting tablet: real 0.016s user 0.012s sys 0.002s
I20260807 22:56:20.482661 29474 heartbeater.cc:503] Master 127.25.94.62:34227 was elected leader, sending a full tablet report...
I20260807 22:56:20.483412 29504 ts_tablet_manager.cc:1428] T 24775f48f02549dab3e902d0663dc6a2 P bb456655a12a41039f76160cfda886ea: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260807 22:56:20.484160 29510 raft_consensus.cc:482] T 24775f48f02549dab3e902d0663dc6a2 P bb456655a12a41039f76160cfda886ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 22:56:20.484443 29510 raft_consensus.cc:504] T 24775f48f02549dab3e902d0663dc6a2 P bb456655a12a41039f76160cfda886ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e5fefbbe4064049919e70cf86eb9757" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 45043 } } peers { permanent_uuid: "856a406170f44f378bfd5638cf75ee7a" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44541 } } peers { permanent_uuid: "bb456655a12a41039f76160cfda886ea" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 45619 } }
I20260807 22:56:20.488061 29503 raft_consensus.cc:348] T 24775f48f02549dab3e902d0663dc6a2 P 856a406170f44f378bfd5638cf75ee7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e5fefbbe4064049919e70cf86eb9757" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 45043 } } peers { permanent_uuid: "856a406170f44f378bfd5638cf75ee7a" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44541 } } peers { permanent_uuid: "bb456655a12a41039f76160cfda886ea" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 45619 } }
I20260807 22:56:20.488628 29503 raft_consensus.cc:374] T 24775f48f02549dab3e902d0663dc6a2 P 856a406170f44f378bfd5638cf75ee7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:56:20.488762 29503 raft_consensus.cc:729] T 24775f48f02549dab3e902d0663dc6a2 P 856a406170f44f378bfd5638cf75ee7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 856a406170f44f378bfd5638cf75ee7a, State: Initialized, Role: FOLLOWER
I20260807 22:56:20.489543 29503 consensus_queue.cc:260] T 24775f48f02549dab3e902d0663dc6a2 P 856a406170f44f378bfd5638cf75ee7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e5fefbbe4064049919e70cf86eb9757" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 45043 } } peers { permanent_uuid: "856a406170f44f378bfd5638cf75ee7a" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44541 } } peers { permanent_uuid: "bb456655a12a41039f76160cfda886ea" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 45619 } }
I20260807 22:56:20.491791 29503 ts_tablet_manager.cc:1428] T 24775f48f02549dab3e902d0663dc6a2 P 856a406170f44f378bfd5638cf75ee7a: Time spent starting tablet: real 0.014s user 0.005s sys 0.008s
I20260807 22:56:20.486614 29510 leader_election.cc:288] T 24775f48f02549dab3e902d0663dc6a2 P bb456655a12a41039f76160cfda886ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9e5fefbbe4064049919e70cf86eb9757 (127.25.94.2:45043), 856a406170f44f378bfd5638cf75ee7a (127.25.94.1:44541)
I20260807 22:56:20.496860 29303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:20.486771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.2:45043 (local address 127.25.94.3:49197)
0807 22:56:20.487450 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0807 22:56:20.487472 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 22:56:20.487620 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:56:20.488121 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:56:20.488126 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:56:20.488153 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:56:20.488415 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:20.488427 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:20.489914 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:20.489919 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:20.492011 (+ 2092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:20.492030 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:20.495929 (+ 3899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:20.495937 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:20.496176 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:20.496227 (+ 51us) client_negotiation.cc:769] Sending connection context
0807 22:56:20.496359 (+ 132us) client_negotiation.cc:240] Negotiation successful
0807 22:56:20.496542 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132}
I20260807 22:56:20.497228 29494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:20.487037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.3:49197 (local address 127.25.94.2:45043)
0807 22:56:20.487246 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:20.487268 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:20.487562 (+ 294us) server_negotiation.cc:407] Connection header received
0807 22:56:20.487759 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:20.487764 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:20.487845 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:20.487941 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:56:20.488597 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:20.489732 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:20.492213 (+ 2481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:20.495533 (+ 3320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:20.495694 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:20.495807 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:20.496735 (+ 928us) server_negotiation.cc:299] Negotiation successful
0807 22:56:20.496887 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260807 22:56:20.498073 29512 raft_consensus.cc:482] T 24775f48f02549dab3e902d0663dc6a2 P 856a406170f44f378bfd5638cf75ee7a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 22:56:20.498283 29193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24775f48f02549dab3e902d0663dc6a2" candidate_uuid: "bb456655a12a41039f76160cfda886ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e5fefbbe4064049919e70cf86eb9757" is_pre_election: true
I20260807 22:56:20.498476 29512 raft_consensus.cc:504] T 24775f48f02549dab3e902d0663dc6a2 P 856a406170f44f378bfd5638cf75ee7a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e5fefbbe4064049919e70cf86eb9757" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 45043 } } peers { permanent_uuid: "856a406170f44f378bfd5638cf75ee7a" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44541 } } peers { permanent_uuid: "bb456655a12a41039f76160cfda886ea" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 45619 } }
I20260807 22:56:20.498880 29193 raft_consensus.cc:2454] T 24775f48f02549dab3e902d0663dc6a2 P 9e5fefbbe4064049919e70cf86eb9757 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb456655a12a41039f76160cfda886ea in term 0.
I20260807 22:56:20.499706 29295 leader_election.cc:302] T 24775f48f02549dab3e902d0663dc6a2 P bb456655a12a41039f76160cfda886ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9e5fefbbe4064049919e70cf86eb9757, bb456655a12a41039f76160cfda886ea; no voters:
I20260807 22:56:20.500568 29512 leader_election.cc:288] T 24775f48f02549dab3e902d0663dc6a2 P 856a406170f44f378bfd5638cf75ee7a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9e5fefbbe4064049919e70cf86eb9757 (127.25.94.2:45043), bb456655a12a41039f76160cfda886ea (127.25.94.3:45619)
I20260807 22:56:20.500854 29510 raft_consensus.cc:2781] T 24775f48f02549dab3e902d0663dc6a2 P bb456655a12a41039f76160cfda886ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 22:56:20.501127 29510 raft_consensus.cc:482] T 24775f48f02549dab3e902d0663dc6a2 P bb456655a12a41039f76160cfda886ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 22:56:20.501360 29510 raft_consensus.cc:3037] T 24775f48f02549dab3e902d0663dc6a2 P bb456655a12a41039f76160cfda886ea [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:56:20.504231 29498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:20.487038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.3:43939 (local address 127.25.94.1:44541)
0807 22:56:20.487412 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:20.487418 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:20.496821 (+ 9403us) server_negotiation.cc:407] Connection header received
0807 22:56:20.497280 (+ 459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:20.497287 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:20.497385 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:20.497470 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:56:20.498169 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:20.499315 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:20.501605 (+ 2290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:20.503003 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:20.503115 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:20.503224 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:20.503850 (+ 626us) server_negotiation.cc:299] Negotiation successful
0807 22:56:20.503983 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261}
I20260807 22:56:20.505203 29511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:20.487235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.1:44541 (local address 127.25.94.3:43939)
0807 22:56:20.496701 (+ 9466us) negotiation.cc:107] Waiting for socket to connect
0807 22:56:20.496724 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0807 22:56:20.497075 (+ 351us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:56:20.497619 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:56:20.497624 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:56:20.497649 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:56:20.498001 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:20.498021 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:20.499463 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:20.499468 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:20.501392 (+ 1924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:20.501413 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:20.503440 (+ 2027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:20.503446 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:20.503639 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:20.503677 (+ 38us) client_negotiation.cc:769] Sending connection context
0807 22:56:20.504632 (+ 955us) client_negotiation.cc:240] Negotiation successful
0807 22:56:20.504822 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9307,"thread_start_us":9103,"threads_started":1}
I20260807 22:56:20.506531 28987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24775f48f02549dab3e902d0663dc6a2" candidate_uuid: "bb456655a12a41039f76160cfda886ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "856a406170f44f378bfd5638cf75ee7a" is_pre_election: true
I20260807 22:56:20.507203 28987 raft_consensus.cc:2454] T 24775f48f02549dab3e902d0663dc6a2 P 856a406170f44f378bfd5638cf75ee7a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb456655a12a41039f76160cfda886ea in term 0.
I20260807 22:56:20.505831 29510 raft_consensus.cc:504] T 24775f48f02549dab3e902d0663dc6a2 P bb456655a12a41039f76160cfda886ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e5fefbbe4064049919e70cf86eb9757" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 45043 } } peers { permanent_uuid: "856a406170f44f378bfd5638cf75ee7a" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44541 } } peers { permanent_uuid: "bb456655a12a41039f76160cfda886ea" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 45619 } }
I20260807 22:56:20.509033 29510 leader_election.cc:288] T 24775f48f02549dab3e902d0663dc6a2 P bb456655a12a41039f76160cfda886ea [CANDIDATE]: Term 1 election: Requested vote from peers 9e5fefbbe4064049919e70cf86eb9757 (127.25.94.2:45043), 856a406170f44f378bfd5638cf75ee7a (127.25.94.1:44541)
I20260807 22:56:20.509079 29193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24775f48f02549dab3e902d0663dc6a2" candidate_uuid: "bb456655a12a41039f76160cfda886ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e5fefbbe4064049919e70cf86eb9757"
I20260807 22:56:20.509155 28987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24775f48f02549dab3e902d0663dc6a2" candidate_uuid: "bb456655a12a41039f76160cfda886ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "856a406170f44f378bfd5638cf75ee7a"
I20260807 22:56:20.509354 29193 raft_consensus.cc:3037] T 24775f48f02549dab3e902d0663dc6a2 P 9e5fefbbe4064049919e70cf86eb9757 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:56:20.509375 28987 raft_consensus.cc:3037] T 24775f48f02549dab3e902d0663dc6a2 P 856a406170f44f378bfd5638cf75ee7a [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:56:20.509495 29494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:20.500892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.1:36851 (local address 127.25.94.2:45043)
0807 22:56:20.501065 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:20.501070 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:20.501780 (+ 710us) server_negotiation.cc:407] Connection header received
0807 22:56:20.502038 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:20.502043 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:20.502117 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:20.502174 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:56:20.503124 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:20.504214 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:20.506771 (+ 2557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:20.507835 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:20.507947 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:20.508031 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:20.508782 (+ 751us) server_negotiation.cc:299] Negotiation successful
0807 22:56:20.508971 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260807 22:56:20.511053 29266 tablet.cc:2369] T 24775f48f02549dab3e902d0663dc6a2 P 9e5fefbbe4064049919e70cf86eb9757: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 22:56:20.512332 29513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:20.500721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.2:45043 (local address 127.25.94.1:36851)
0807 22:56:20.501691 (+ 970us) negotiation.cc:107] Waiting for socket to connect
0807 22:56:20.501710 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 22:56:20.501878 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:56:20.502382 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:56:20.502387 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:56:20.502465 (+ 78us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:56:20.502944 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:20.502957 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:20.504469 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:20.504477 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:20.506605 (+ 2128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:20.506618 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:20.508401 (+ 1783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:20.508406 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:20.508594 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:20.508637 (+ 43us) client_negotiation.cc:769] Sending connection context
0807 22:56:20.509123 (+ 486us) client_negotiation.cc:240] Negotiation successful
0807 22:56:20.511979 (+ 2856us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":835,"thread_start_us":662,"threads_started":1}
I20260807 22:56:20.512770 28987 raft_consensus.cc:2454] T 24775f48f02549dab3e902d0663dc6a2 P 856a406170f44f378bfd5638cf75ee7a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb456655a12a41039f76160cfda886ea in term 1.
I20260807 22:56:20.512830 29193 raft_consensus.cc:2454] T 24775f48f02549dab3e902d0663dc6a2 P 9e5fefbbe4064049919e70cf86eb9757 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb456655a12a41039f76160cfda886ea in term 1.
I20260807 22:56:20.513274 29294 leader_election.cc:302] T 24775f48f02549dab3e902d0663dc6a2 P bb456655a12a41039f76160cfda886ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 856a406170f44f378bfd5638cf75ee7a, bb456655a12a41039f76160cfda886ea; no voters:
I20260807 22:56:20.513628 29510 raft_consensus.cc:2781] T 24775f48f02549dab3e902d0663dc6a2 P bb456655a12a41039f76160cfda886ea [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:56:20.513970 29193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24775f48f02549dab3e902d0663dc6a2" candidate_uuid: "856a406170f44f378bfd5638cf75ee7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e5fefbbe4064049919e70cf86eb9757" is_pre_election: true
I20260807 22:56:20.514156 29510 raft_consensus.cc:686] T 24775f48f02549dab3e902d0663dc6a2 P bb456655a12a41039f76160cfda886ea [term 1 LEADER]: Becoming Leader. State: Replica: bb456655a12a41039f76160cfda886ea, State: Running, Role: LEADER
I20260807 22:56:20.514397 29193 raft_consensus.cc:2379] T 24775f48f02549dab3e902d0663dc6a2 P 9e5fefbbe4064049919e70cf86eb9757 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 856a406170f44f378bfd5638cf75ee7a in current term 1: Already voted for candidate bb456655a12a41039f76160cfda886ea in this term.
I20260807 22:56:20.514775 29510 consensus_queue.cc:237] T 24775f48f02549dab3e902d0663dc6a2 P bb456655a12a41039f76160cfda886ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e5fefbbe4064049919e70cf86eb9757" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 45043 } } peers { permanent_uuid: "856a406170f44f378bfd5638cf75ee7a" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44541 } } peers { permanent_uuid: "bb456655a12a41039f76160cfda886ea" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 45619 } }
I20260807 22:56:20.517336 29515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:20.501835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.3:45619 (local address 127.25.94.1:57557)
0807 22:56:20.510191 (+ 8356us) negotiation.cc:107] Waiting for socket to connect
0807 22:56:20.510211 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 22:56:20.510394 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:56:20.510887 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:56:20.510891 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:56:20.510923 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:56:20.511418 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:20.511430 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:20.512889 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:20.512895 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:20.514944 (+ 2049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:20.514963 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:20.516600 (+ 1637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:20.516605 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:20.516739 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:20.516763 (+ 24us) client_negotiation.cc:769] Sending connection context
0807 22:56:20.516841 (+ 78us) client_negotiation.cc:240] Negotiation successful
0807 22:56:20.516934 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8209,"thread_start_us":11503,"threads_started":1}
I20260807 22:56:20.517408 29501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:20.502193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.1:57557 (local address 127.25.94.3:45619)
0807 22:56:20.502415 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:20.502421 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:20.510293 (+ 7872us) server_negotiation.cc:407] Connection header received
0807 22:56:20.510545 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:20.510550 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:20.510620 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:20.510676 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:56:20.511635 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:20.512678 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:20.515147 (+ 2469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:20.516388 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:20.516510 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:20.516628 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:20.516893 (+ 265us) server_negotiation.cc:299] Negotiation successful
0807 22:56:20.517041 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260807 22:56:20.519461 29407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24775f48f02549dab3e902d0663dc6a2" candidate_uuid: "856a406170f44f378bfd5638cf75ee7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb456655a12a41039f76160cfda886ea" is_pre_election: true
I20260807 22:56:20.520579 28871 leader_election.cc:302] T 24775f48f02549dab3e902d0663dc6a2 P 856a406170f44f378bfd5638cf75ee7a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 856a406170f44f378bfd5638cf75ee7a; no voters: 9e5fefbbe4064049919e70cf86eb9757, bb456655a12a41039f76160cfda886ea
I20260807 22:56:20.520323 28728 catalog_manager.cc:5495] T 24775f48f02549dab3e902d0663dc6a2 P bb456655a12a41039f76160cfda886ea reported cstate change: term changed from 0 to 1, leader changed from <none> to bb456655a12a41039f76160cfda886ea (127.25.94.3). New cstate: current_term: 1 leader_uuid: "bb456655a12a41039f76160cfda886ea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e5fefbbe4064049919e70cf86eb9757" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 45043 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "856a406170f44f378bfd5638cf75ee7a" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44541 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb456655a12a41039f76160cfda886ea" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 45619 } health_report { overall_health: HEALTHY } } }
I20260807 22:56:20.521258 29512 raft_consensus.cc:2726] T 24775f48f02549dab3e902d0663dc6a2 P 856a406170f44f378bfd5638cf75ee7a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
Aug 07 22:56:20 dist-test-slave-z8x9 krb5kdc[28646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143378, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.62@KRBTEST.COM for kudu/127.25.94.62@KRBTEST.COM
I20260807 22:56:20.569423 29301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:20.555728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42584 (local address 127.25.94.62:34227)
0807 22:56:20.555919 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:20.555924 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:20.556503 (+ 579us) server_negotiation.cc:407] Connection header received
0807 22:56:20.556706 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:20.556710 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:20.556778 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:20.556881 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:56:20.558164 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:20.558929 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:20.560157 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:20.560585 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:20.560748 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:20.564052 (+ 3304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:56:20.564057 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:56:20.564061 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:56:20.564091 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:56:20.566615 (+ 2524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:20.567361 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:20.567368 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:20.567372 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:20.567511 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:20.567878 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:20.567885 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:20.567888 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:20.568155 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:56:20.568310 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:20.568710 (+ 400us) server_negotiation.cc:299] Negotiation successful
0807 22:56:20.569053 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":143,"server-negotiator.queue_time_us":71}
I20260807 22:56:20.569458 29523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:20.555732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:34227 (local address 127.0.0.1:42584)
0807 22:56:20.556382 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0807 22:56:20.556398 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 22:56:20.556579 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:56:20.556905 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:56:20.556909 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:56:20.557721 (+ 812us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:56:20.557992 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:20.558002 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:20.559061 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:20.559069 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:20.560015 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:20.560026 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:20.560774 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:20.560780 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:20.560933 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:20.561667 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:56:20.561688 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:56:20.563865 (+ 2177us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:56:20.566770 (+ 2905us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:56:20.566784 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:56:20.566799 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:56:20.567215 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:56:20.567622 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:56:20.567628 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:56:20.567631 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:56:20.567771 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:56:20.568295 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:56:20.568303 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:56:20.568468 (+ 165us) client_negotiation.cc:769] Sending connection context
0807 22:56:20.568634 (+ 166us) client_negotiation.cc:240] Negotiation successful
0807 22:56:20.568924 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":343,"threads_started":1}
I20260807 22:56:20.575042 28728 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.25.94.62@KRBTEST.COM'} at 127.0.0.1:42584:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260807 22:56:20.576304 28728 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 22:56:20.583052 28950 tablet_service.cc:1467] Processing CreateTablet for tablet 710c46a2c0d846fbb57f14284ad73d39 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7e510070cbe34840b433ac9cd62bccc7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260807 22:56:20.583187 29163 tablet_service.cc:1467] Processing CreateTablet for tablet 710c46a2c0d846fbb57f14284ad73d39 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7e510070cbe34840b433ac9cd62bccc7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260807 22:56:20.583349 29362 tablet_service.cc:1467] Processing CreateTablet for tablet 710c46a2c0d846fbb57f14284ad73d39 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7e510070cbe34840b433ac9cd62bccc7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260807 22:56:20.584110 29362 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 710c46a2c0d846fbb57f14284ad73d39. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:56:20.584110 28950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 710c46a2c0d846fbb57f14284ad73d39. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:56:20.584146 29163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 710c46a2c0d846fbb57f14284ad73d39. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:56:20.590420 29502 tablet_bootstrap.cc:492] T 710c46a2c0d846fbb57f14284ad73d39 P 9e5fefbbe4064049919e70cf86eb9757: Bootstrap starting.
I20260807 22:56:20.591446 29503 tablet_bootstrap.cc:492] T 710c46a2c0d846fbb57f14284ad73d39 P 856a406170f44f378bfd5638cf75ee7a: Bootstrap starting.
I20260807 22:56:20.591506 29504 tablet_bootstrap.cc:492] T 710c46a2c0d846fbb57f14284ad73d39 P bb456655a12a41039f76160cfda886ea: Bootstrap starting.
I20260807 22:56:20.592197 29502 tablet_bootstrap.cc:654] T 710c46a2c0d846fbb57f14284ad73d39 P 9e5fefbbe4064049919e70cf86eb9757: Neither blocks nor log segments found. Creating new log.
I20260807 22:56:20.592916 29503 tablet_bootstrap.cc:654] T 710c46a2c0d846fbb57f14284ad73d39 P 856a406170f44f378bfd5638cf75ee7a: Neither blocks nor log segments found. Creating new log.
I20260807 22:56:20.593380 29504 tablet_bootstrap.cc:654] T 710c46a2c0d846fbb57f14284ad73d39 P bb456655a12a41039f76160cfda886ea: Neither blocks nor log segments found. Creating new log.
I20260807 22:56:20.594702 29502 tablet_bootstrap.cc:492] T 710c46a2c0d846fbb57f14284ad73d39 P 9e5fefbbe4064049919e70cf86eb9757: No bootstrap required, opened a new log
I20260807 22:56:20.594848 29502 ts_tablet_manager.cc:1397] T 710c46a2c0d846fbb57f14284ad73d39 P 9e5fefbbe4064049919e70cf86eb9757: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260807 22:56:20.595633 29502 raft_consensus.cc:348] T 710c46a2c0d846fbb57f14284ad73d39 P 9e5fefbbe4064049919e70cf86eb9757 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "856a406170f44f378bfd5638cf75ee7a" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44541 } } peers { permanent_uuid: "9e5fefbbe4064049919e70cf86eb9757" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 45043 } } peers { permanent_uuid: "bb456655a12a41039f76160cfda886ea" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 45619 } }
I20260807 22:56:20.595958 29502 raft_consensus.cc:374] T 710c46a2c0d846fbb57f14284ad73d39 P 9e5fefbbe4064049919e70cf86eb9757 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260807 22:56:20.595976 29475 tablet.cc:2369] T 24775f48f02549dab3e902d0663dc6a2 P bb456655a12a41039f76160cfda886ea: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 22:56:20.596073 29502 raft_consensus.cc:729] T 710c46a2c0d846fbb57f14284ad73d39 P 9e5fefbbe4064049919e70cf86eb9757 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e5fefbbe4064049919e70cf86eb9757, State: Initialized, Role: FOLLOWER
I20260807 22:56:20.596326 29502 consensus_queue.cc:260] T 710c46a2c0d846fbb57f14284ad73d39 P 9e5fefbbe4064049919e70cf86eb9757 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "856a406170f44f378bfd5638cf75ee7a" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44541 } } peers { permanent_uuid: "9e5fefbbe4064049919e70cf86eb9757" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 45043 } } peers { permanent_uuid: "bb456655a12a41039f76160cfda886ea" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 45619 } }
I20260807 22:56:20.596912 29502 ts_tablet_manager.cc:1428] T 710c46a2c0d846fbb57f14284ad73d39 P 9e5fefbbe4064049919e70cf86eb9757: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260807 22:56:20.597004 29509 tablet_replica.cc:439] T 710c46a2c0d846fbb57f14284ad73d39 P 9e5fefbbe4064049919e70cf86eb9757: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "856a406170f44f378bfd5638cf75ee7a" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44541 } } peers { permanent_uuid: "9e5fefbbe4064049919e70cf86eb9757" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 45043 } } peers { permanent_uuid: "bb456655a12a41039f76160cfda886ea" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 45619 } } }
I20260807 22:56:20.597460 29509 tablet_replica.cc:442] T 710c46a2c0d846fbb57f14284ad73d39 P 9e5fefbbe4064049919e70cf86eb9757: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 22:56:20.598344 29504 tablet_bootstrap.cc:492] T 710c46a2c0d846fbb57f14284ad73d39 P bb456655a12a41039f76160cfda886ea: No bootstrap required, opened a new log
I20260807 22:56:20.598526 29504 ts_tablet_manager.cc:1397] T 710c46a2c0d846fbb57f14284ad73d39 P bb456655a12a41039f76160cfda886ea: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.002s
I20260807 22:56:20.598692 29503 tablet_bootstrap.cc:492] T 710c46a2c0d846fbb57f14284ad73d39 P 856a406170f44f378bfd5638cf75ee7a: No bootstrap required, opened a new log
I20260807 22:56:20.598856 29503 ts_tablet_manager.cc:1397] T 710c46a2c0d846fbb57f14284ad73d39 P 856a406170f44f378bfd5638cf75ee7a: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s
I20260807 22:56:20.599421 29504 raft_consensus.cc:348] T 710c46a2c0d846fbb57f14284ad73d39 P bb456655a12a41039f76160cfda886ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "856a406170f44f378bfd5638cf75ee7a" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44541 } } peers { permanent_uuid: "9e5fefbbe4064049919e70cf86eb9757" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 45043 } } peers { permanent_uuid: "bb456655a12a41039f76160cfda886ea" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 45619 } }
I20260807 22:56:20.599681 29504 raft_consensus.cc:374] T 710c46a2c0d846fbb57f14284ad73d39 P bb456655a12a41039f76160cfda886ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:56:20.599792 29504 raft_consensus.cc:729] T 710c46a2c0d846fbb57f14284ad73d39 P bb456655a12a41039f76160cfda886ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb456655a12a41039f76160cfda886ea, State: Initialized, Role: FOLLOWER
I20260807 22:56:20.599946 29503 raft_consensus.cc:348] T 710c46a2c0d846fbb57f14284ad73d39 P 856a406170f44f378bfd5638cf75ee7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "856a406170f44f378bfd5638cf75ee7a" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44541 } } peers { permanent_uuid: "9e5fefbbe4064049919e70cf86eb9757" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 45043 } } peers { permanent_uuid: "bb456655a12a41039f76160cfda886ea" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 45619 } }
I20260807 22:56:20.600174 29503 raft_consensus.cc:374] T 710c46a2c0d846fbb57f14284ad73d39 P 856a406170f44f378bfd5638cf75ee7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:56:20.600067 29504 consensus_queue.cc:260] T 710c46a2c0d846fbb57f14284ad73d39 P bb456655a12a41039f76160cfda886ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "856a406170f44f378bfd5638cf75ee7a" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44541 } } peers { permanent_uuid: "9e5fefbbe4064049919e70cf86eb9757" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 45043 } } peers { permanent_uuid: "bb456655a12a41039f76160cfda886ea" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 45619 } }
I20260807 22:56:20.600299 29503 raft_consensus.cc:729] T 710c46a2c0d846fbb57f14284ad73d39 P 856a406170f44f378bfd5638cf75ee7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 856a406170f44f378bfd5638cf75ee7a, State: Initialized, Role: FOLLOWER
I20260807 22:56:20.600539 29503 consensus_queue.cc:260] T 710c46a2c0d846fbb57f14284ad73d39 P 856a406170f44f378bfd5638cf75ee7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "856a406170f44f378bfd5638cf75ee7a" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44541 } } peers { permanent_uuid: "9e5fefbbe4064049919e70cf86eb9757" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 45043 } } peers { permanent_uuid: "bb456655a12a41039f76160cfda886ea" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 45619 } }
I20260807 22:56:20.600503 29510 tablet_replica.cc:439] T 710c46a2c0d846fbb57f14284ad73d39 P bb456655a12a41039f76160cfda886ea: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "856a406170f44f378bfd5638cf75ee7a" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44541 } } peers { permanent_uuid: "9e5fefbbe4064049919e70cf86eb9757" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 45043 } } peers { permanent_uuid: "bb456655a12a41039f76160cfda886ea" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 45619 } } }
I20260807 22:56:20.600685 29504 ts_tablet_manager.cc:1428] T 710c46a2c0d846fbb57f14284ad73d39 P bb456655a12a41039f76160cfda886ea: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260807 22:56:20.600804 29510 tablet_replica.cc:442] T 710c46a2c0d846fbb57f14284ad73d39 P bb456655a12a41039f76160cfda886ea: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 22:56:20.600970 29503 ts_tablet_manager.cc:1428] T 710c46a2c0d846fbb57f14284ad73d39 P 856a406170f44f378bfd5638cf75ee7a: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260807 22:56:20.601452 29512 tablet_replica.cc:439] T 710c46a2c0d846fbb57f14284ad73d39 P 856a406170f44f378bfd5638cf75ee7a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "856a406170f44f378bfd5638cf75ee7a" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44541 } } peers { permanent_uuid: "9e5fefbbe4064049919e70cf86eb9757" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 45043 } } peers { permanent_uuid: "bb456655a12a41039f76160cfda886ea" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 45619 } } }
I20260807 22:56:20.601707 29512 tablet_replica.cc:442] T 710c46a2c0d846fbb57f14284ad73d39 P 856a406170f44f378bfd5638cf75ee7a: This TxnStatusTablet replica's current role is: FOLLOWER
W20260807 22:56:20.717940 29061 tablet.cc:2369] T 24775f48f02549dab3e902d0663dc6a2 P 856a406170f44f378bfd5638cf75ee7a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 22:56:20.722463 29509 raft_consensus.cc:482] T 710c46a2c0d846fbb57f14284ad73d39 P 9e5fefbbe4064049919e70cf86eb9757 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 22:56:20.722694 29509 raft_consensus.cc:504] T 710c46a2c0d846fbb57f14284ad73d39 P 9e5fefbbe4064049919e70cf86eb9757 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "856a406170f44f378bfd5638cf75ee7a" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44541 } } peers { permanent_uuid: "9e5fefbbe4064049919e70cf86eb9757" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 45043 } } peers { permanent_uuid: "bb456655a12a41039f76160cfda886ea" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 45619 } }
I20260807 22:56:20.724225 29509 leader_election.cc:288] T 710c46a2c0d846fbb57f14284ad73d39 P 9e5fefbbe4064049919e70cf86eb9757 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 856a406170f44f378bfd5638cf75ee7a (127.25.94.1:44541), bb456655a12a41039f76160cfda886ea (127.25.94.3:45619)
I20260807 22:56:20.733770 29501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:20.724881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.2:34123 (local address 127.25.94.3:45619)
0807 22:56:20.725102 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:20.725110 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:20.725389 (+ 279us) server_negotiation.cc:407] Connection header received
0807 22:56:20.725729 (+ 340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:20.725736 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:20.725817 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:20.725885 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:56:20.726687 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:20.727853 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:20.731087 (+ 3234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:20.732348 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:20.732506 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:20.732612 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:20.733306 (+ 694us) server_negotiation.cc:299] Negotiation successful
0807 22:56:20.733448 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260807 22:56:20.733829 29498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:20.725458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.2:32769 (local address 127.25.94.1:44541)
0807 22:56:20.725662 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:20.725670 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:20.726037 (+ 367us) server_negotiation.cc:407] Connection header received
0807 22:56:20.726184 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:20.726187 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:20.726248 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:20.726346 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:56:20.727002 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:20.728100 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:20.731087 (+ 2987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:20.732433 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:20.732549 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:20.732641 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:20.733364 (+ 723us) server_negotiation.cc:299] Negotiation successful
0807 22:56:20.733502 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260807 22:56:20.733949 29532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:20.724742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.1:44541 (local address 127.25.94.2:32769)
0807 22:56:20.725926 (+ 1184us) negotiation.cc:107] Waiting for socket to connect
0807 22:56:20.725944 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 22:56:20.726063 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:56:20.726400 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:56:20.726404 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:56:20.726433 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:56:20.726827 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:20.726835 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:20.728254 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:20.728261 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:20.730885 (+ 2624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:20.730901 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:20.732757 (+ 1856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:20.732768 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:20.733015 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:20.733186 (+ 171us) client_negotiation.cc:769] Sending connection context
0807 22:56:20.733302 (+ 116us) client_negotiation.cc:240] Negotiation successful
0807 22:56:20.733455 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1064,"mutex_wait_us":302,"thread_start_us":826,"threads_started":1}
I20260807 22:56:20.734010 29530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:20.724495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.3:45619 (local address 127.25.94.2:34123)
0807 22:56:20.725284 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0807 22:56:20.725306 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 22:56:20.725524 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:56:20.726099 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:56:20.726105 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:56:20.726131 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:56:20.726504 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:20.726518 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:20.728010 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:20.728015 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:20.730889 (+ 2874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:20.730901 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:20.732754 (+ 1853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:20.732761 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:20.733014 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:20.733119 (+ 105us) client_negotiation.cc:769] Sending connection context
0807 22:56:20.733296 (+ 177us) client_negotiation.cc:240] Negotiation successful
0807 22:56:20.733448 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":594,"mutex_wait_us":147,"thread_start_us":426,"threads_started":1}
I20260807 22:56:20.734745 29407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "710c46a2c0d846fbb57f14284ad73d39" candidate_uuid: "9e5fefbbe4064049919e70cf86eb9757" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb456655a12a41039f76160cfda886ea" is_pre_election: true
I20260807 22:56:20.734750 28987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "710c46a2c0d846fbb57f14284ad73d39" candidate_uuid: "9e5fefbbe4064049919e70cf86eb9757" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "856a406170f44f378bfd5638cf75ee7a" is_pre_election: true
I20260807 22:56:20.735046 28987 raft_consensus.cc:2454] T 710c46a2c0d846fbb57f14284ad73d39 P 856a406170f44f378bfd5638cf75ee7a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e5fefbbe4064049919e70cf86eb9757 in term 0.
I20260807 22:56:20.735112 29407 raft_consensus.cc:2454] T 710c46a2c0d846fbb57f14284ad73d39 P bb456655a12a41039f76160cfda886ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e5fefbbe4064049919e70cf86eb9757 in term 0.
I20260807 22:56:20.735648 29094 leader_election.cc:302] T 710c46a2c0d846fbb57f14284ad73d39 P 9e5fefbbe4064049919e70cf86eb9757 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 856a406170f44f378bfd5638cf75ee7a, 9e5fefbbe4064049919e70cf86eb9757, bb456655a12a41039f76160cfda886ea; no voters:
I20260807 22:56:20.736184 29509 raft_consensus.cc:2781] T 710c46a2c0d846fbb57f14284ad73d39 P 9e5fefbbe4064049919e70cf86eb9757 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 22:56:20.736371 29509 raft_consensus.cc:482] T 710c46a2c0d846fbb57f14284ad73d39 P 9e5fefbbe4064049919e70cf86eb9757 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 22:56:20.736487 29509 raft_consensus.cc:3037] T 710c46a2c0d846fbb57f14284ad73d39 P 9e5fefbbe4064049919e70cf86eb9757 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:56:20.738839 29509 raft_consensus.cc:504] T 710c46a2c0d846fbb57f14284ad73d39 P 9e5fefbbe4064049919e70cf86eb9757 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "856a406170f44f378bfd5638cf75ee7a" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44541 } } peers { permanent_uuid: "9e5fefbbe4064049919e70cf86eb9757" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 45043 } } peers { permanent_uuid: "bb456655a12a41039f76160cfda886ea" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 45619 } }
I20260807 22:56:20.739495 29509 leader_election.cc:288] T 710c46a2c0d846fbb57f14284ad73d39 P 9e5fefbbe4064049919e70cf86eb9757 [CANDIDATE]: Term 1 election: Requested vote from peers 856a406170f44f378bfd5638cf75ee7a (127.25.94.1:44541), bb456655a12a41039f76160cfda886ea (127.25.94.3:45619)
I20260807 22:56:20.740013 28987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "710c46a2c0d846fbb57f14284ad73d39" candidate_uuid: "9e5fefbbe4064049919e70cf86eb9757" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "856a406170f44f378bfd5638cf75ee7a"
I20260807 22:56:20.740041 29407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "710c46a2c0d846fbb57f14284ad73d39" candidate_uuid: "9e5fefbbe4064049919e70cf86eb9757" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb456655a12a41039f76160cfda886ea"
I20260807 22:56:20.740275 29407 raft_consensus.cc:3037] T 710c46a2c0d846fbb57f14284ad73d39 P bb456655a12a41039f76160cfda886ea [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:56:20.740275 28987 raft_consensus.cc:3037] T 710c46a2c0d846fbb57f14284ad73d39 P 856a406170f44f378bfd5638cf75ee7a [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:56:20.743342 29407 raft_consensus.cc:2454] T 710c46a2c0d846fbb57f14284ad73d39 P bb456655a12a41039f76160cfda886ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e5fefbbe4064049919e70cf86eb9757 in term 1.
I20260807 22:56:20.743347 28987 raft_consensus.cc:2454] T 710c46a2c0d846fbb57f14284ad73d39 P 856a406170f44f378bfd5638cf75ee7a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e5fefbbe4064049919e70cf86eb9757 in term 1.
I20260807 22:56:20.743796 29095 leader_election.cc:302] T 710c46a2c0d846fbb57f14284ad73d39 P 9e5fefbbe4064049919e70cf86eb9757 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9e5fefbbe4064049919e70cf86eb9757, bb456655a12a41039f76160cfda886ea; no voters:
I20260807 22:56:20.744150 29509 raft_consensus.cc:2781] T 710c46a2c0d846fbb57f14284ad73d39 P 9e5fefbbe4064049919e70cf86eb9757 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:56:20.744728 29509 raft_consensus.cc:686] T 710c46a2c0d846fbb57f14284ad73d39 P 9e5fefbbe4064049919e70cf86eb9757 [term 1 LEADER]: Becoming Leader. State: Replica: 9e5fefbbe4064049919e70cf86eb9757, State: Running, Role: LEADER
I20260807 22:56:20.745359 29509 consensus_queue.cc:237] T 710c46a2c0d846fbb57f14284ad73d39 P 9e5fefbbe4064049919e70cf86eb9757 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "856a406170f44f378bfd5638cf75ee7a" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44541 } } peers { permanent_uuid: "9e5fefbbe4064049919e70cf86eb9757" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 45043 } } peers { permanent_uuid: "bb456655a12a41039f76160cfda886ea" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 45619 } }
I20260807 22:56:20.748234 29533 tablet_replica.cc:439] T 710c46a2c0d846fbb57f14284ad73d39 P 9e5fefbbe4064049919e70cf86eb9757: TxnStatusTablet state changed. Reason: New leader 9e5fefbbe4064049919e70cf86eb9757. Latest consensus state: current_term: 1 leader_uuid: "9e5fefbbe4064049919e70cf86eb9757" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "856a406170f44f378bfd5638cf75ee7a" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44541 } } peers { permanent_uuid: "9e5fefbbe4064049919e70cf86eb9757" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 45043 } } peers { permanent_uuid: "bb456655a12a41039f76160cfda886ea" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 45619 } } }
I20260807 22:56:20.748567 29533 tablet_replica.cc:442] T 710c46a2c0d846fbb57f14284ad73d39 P 9e5fefbbe4064049919e70cf86eb9757: This TxnStatusTablet replica's current role is: LEADER
I20260807 22:56:20.749404 29538 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260807 22:56:20.749688 29538 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260807 22:56:20.750223 28727 catalog_manager.cc:5495] T 710c46a2c0d846fbb57f14284ad73d39 P 9e5fefbbe4064049919e70cf86eb9757 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9e5fefbbe4064049919e70cf86eb9757 (127.25.94.2). New cstate: current_term: 1 leader_uuid: "9e5fefbbe4064049919e70cf86eb9757" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "856a406170f44f378bfd5638cf75ee7a" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44541 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e5fefbbe4064049919e70cf86eb9757" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 45043 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb456655a12a41039f76160cfda886ea" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 45619 } health_report { overall_health: UNKNOWN } } }
I20260807 22:56:20.792781 29494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:20.786248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55186 (local address 127.25.94.2:45043)
0807 22:56:20.786427 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:20.786431 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:20.786474 (+ 43us) server_negotiation.cc:407] Connection header received
0807 22:56:20.786821 (+ 347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:20.786825 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:20.786913 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:20.787032 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 22:56:20.787484 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:20.788243 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:20.790236 (+ 1993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:20.790777 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:20.790873 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:20.791461 (+ 588us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 22:56:20.792040 (+ 579us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 22:56:20.792146 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:20.792365 (+ 219us) server_negotiation.cc:299] Negotiation successful
0807 22:56:20.792463 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260807 22:56:20.792805 29523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:20.786081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.2:45043 (local address 127.0.0.1:55186)
0807 22:56:20.786389 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0807 22:56:20.786412 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0807 22:56:20.786605 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:56:20.787088 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:56:20.787094 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:56:20.787108 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0807 22:56:20.787334 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:20.787347 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:20.788414 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:20.788420 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:20.790089 (+ 1669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:20.790107 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:20.790947 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:20.790955 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:20.791183 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:20.791257 (+ 74us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0807 22:56:20.792157 (+ 900us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0807 22:56:20.792181 (+ 24us) client_negotiation.cc:769] Sending connection context
0807 22:56:20.792287 (+ 106us) client_negotiation.cc:240] Negotiation successful
0807 22:56:20.792481 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260807 22:56:20.800617 29407 raft_consensus.cc:1261] T 710c46a2c0d846fbb57f14284ad73d39 P bb456655a12a41039f76160cfda886ea [term 1 FOLLOWER]: Refusing update from remote peer 9e5fefbbe4064049919e70cf86eb9757: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 22:56:20.800940 28987 raft_consensus.cc:1261] T 710c46a2c0d846fbb57f14284ad73d39 P 856a406170f44f378bfd5638cf75ee7a [term 1 FOLLOWER]: Refusing update from remote peer 9e5fefbbe4064049919e70cf86eb9757: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 22:56:20.801551 29533 consensus_queue.cc:1035] T 710c46a2c0d846fbb57f14284ad73d39 P 9e5fefbbe4064049919e70cf86eb9757 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb456655a12a41039f76160cfda886ea" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 45619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 22:56:20.801913 29509 consensus_queue.cc:1035] T 710c46a2c0d846fbb57f14284ad73d39 P 9e5fefbbe4064049919e70cf86eb9757 [LEADER]: Connected to new peer: Peer: permanent_uuid: "856a406170f44f378bfd5638cf75ee7a" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 22:56:20.816666 29510 tablet_replica.cc:439] T 710c46a2c0d846fbb57f14284ad73d39 P bb456655a12a41039f76160cfda886ea: TxnStatusTablet state changed. Reason: New leader 9e5fefbbe4064049919e70cf86eb9757. Latest consensus state: current_term: 1 leader_uuid: "9e5fefbbe4064049919e70cf86eb9757" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "856a406170f44f378bfd5638cf75ee7a" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44541 } } peers { permanent_uuid: "9e5fefbbe4064049919e70cf86eb9757" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 45043 } } peers { permanent_uuid: "bb456655a12a41039f76160cfda886ea" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 45619 } } }
I20260807 22:56:20.817034 29510 tablet_replica.cc:442] T 710c46a2c0d846fbb57f14284ad73d39 P bb456655a12a41039f76160cfda886ea: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 22:56:20.818014 29541 mvcc.cc:205] Tried to move back new op lower bound from 7316043287750004736 to 7316043287536865280. Current Snapshot: MvccSnapshot[applied={T|T < 7316043287750004736}]
I20260807 22:56:20.818398 29509 tablet_replica.cc:439] T 710c46a2c0d846fbb57f14284ad73d39 P 9e5fefbbe4064049919e70cf86eb9757: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9e5fefbbe4064049919e70cf86eb9757" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "856a406170f44f378bfd5638cf75ee7a" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44541 } } peers { permanent_uuid: "9e5fefbbe4064049919e70cf86eb9757" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 45043 } } peers { permanent_uuid: "bb456655a12a41039f76160cfda886ea" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 45619 } } }
I20260807 22:56:20.818699 29509 tablet_replica.cc:442] T 710c46a2c0d846fbb57f14284ad73d39 P 9e5fefbbe4064049919e70cf86eb9757: This TxnStatusTablet replica's current role is: LEADER
I20260807 22:56:20.819691 29543 mvcc.cc:205] Tried to move back new op lower bound from 7316043287750004736 to 7316043287536865280. Current Snapshot: MvccSnapshot[applied={T|T < 7316043287750004736}]
I20260807 22:56:20.822630 29510 tablet_replica.cc:439] T 710c46a2c0d846fbb57f14284ad73d39 P bb456655a12a41039f76160cfda886ea: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9e5fefbbe4064049919e70cf86eb9757" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "856a406170f44f378bfd5638cf75ee7a" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44541 } } peers { permanent_uuid: "9e5fefbbe4064049919e70cf86eb9757" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 45043 } } peers { permanent_uuid: "bb456655a12a41039f76160cfda886ea" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 45619 } } }
I20260807 22:56:20.823005 29510 tablet_replica.cc:442] T 710c46a2c0d846fbb57f14284ad73d39 P bb456655a12a41039f76160cfda886ea: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 22:56:20.828655 29512 tablet_replica.cc:439] T 710c46a2c0d846fbb57f14284ad73d39 P 856a406170f44f378bfd5638cf75ee7a: TxnStatusTablet state changed. Reason: New leader 9e5fefbbe4064049919e70cf86eb9757. Latest consensus state: current_term: 1 leader_uuid: "9e5fefbbe4064049919e70cf86eb9757" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "856a406170f44f378bfd5638cf75ee7a" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44541 } } peers { permanent_uuid: "9e5fefbbe4064049919e70cf86eb9757" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 45043 } } peers { permanent_uuid: "bb456655a12a41039f76160cfda886ea" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 45619 } } }
I20260807 22:56:20.828987 29512 tablet_replica.cc:442] T 710c46a2c0d846fbb57f14284ad73d39 P 856a406170f44f378bfd5638cf75ee7a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 22:56:20.834510 29533 tablet_replica.cc:439] T 710c46a2c0d846fbb57f14284ad73d39 P 9e5fefbbe4064049919e70cf86eb9757: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9e5fefbbe4064049919e70cf86eb9757" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "856a406170f44f378bfd5638cf75ee7a" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44541 } } peers { permanent_uuid: "9e5fefbbe4064049919e70cf86eb9757" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 45043 } } peers { permanent_uuid: "bb456655a12a41039f76160cfda886ea" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 45619 } } }
I20260807 22:56:20.834821 29533 tablet_replica.cc:442] T 710c46a2c0d846fbb57f14284ad73d39 P 9e5fefbbe4064049919e70cf86eb9757: This TxnStatusTablet replica's current role is: LEADER
I20260807 22:56:20.848240 29501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:20.837711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59698 (local address 127.25.94.3:45619)
0807 22:56:20.837885 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:20.837892 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:20.839169 (+ 1277us) server_negotiation.cc:407] Connection header received
0807 22:56:20.839301 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:20.839310 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:20.839482 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:20.839664 (+ 182us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 22:56:20.840389 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:20.841813 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:20.844370 (+ 2557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:20.845292 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:20.845445 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:20.846266 (+ 821us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 22:56:20.847096 (+ 830us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 22:56:20.847229 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:20.847699 (+ 470us) server_negotiation.cc:299] Negotiation successful
0807 22:56:20.847830 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260807 22:56:20.849748 29512 tablet_replica.cc:439] T 710c46a2c0d846fbb57f14284ad73d39 P 856a406170f44f378bfd5638cf75ee7a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9e5fefbbe4064049919e70cf86eb9757" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "856a406170f44f378bfd5638cf75ee7a" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44541 } } peers { permanent_uuid: "9e5fefbbe4064049919e70cf86eb9757" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 45043 } } peers { permanent_uuid: "bb456655a12a41039f76160cfda886ea" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 45619 } } }
I20260807 22:56:20.850041 29512 tablet_replica.cc:442] T 710c46a2c0d846fbb57f14284ad73d39 P 856a406170f44f378bfd5638cf75ee7a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 22:56:20.865749 29494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:20.859567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.3:49049 (local address 127.25.94.2:45043)
0807 22:56:20.859749 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:20.859754 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:20.860153 (+ 399us) server_negotiation.cc:407] Connection header received
0807 22:56:20.860377 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:20.860381 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:20.860449 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:20.860527 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 22:56:20.861058 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:20.861871 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:20.863842 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:20.864319 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:20.864409 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:20.864986 (+ 577us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 22:56:20.865090 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 22:56:20.865155 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:20.865398 (+ 243us) server_negotiation.cc:299] Negotiation successful
0807 22:56:20.865484 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260807 22:56:20.865875 29559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:20.859384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.2:45043 (local address 127.25.94.3:49049)
0807 22:56:20.860042 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0807 22:56:20.860064 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 22:56:20.860222 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:56:20.860574 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:56:20.860582 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:56:20.860598 (+ 16us) client_negotiation.cc:189] Negotiated authn=TOKEN
0807 22:56:20.860906 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:20.860917 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:20.862011 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:20.862018 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:20.863693 (+ 1675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:20.863711 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:20.864505 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:20.864511 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:20.864707 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:20.864781 (+ 74us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0807 22:56:20.865220 (+ 439us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0807 22:56:20.865234 (+ 14us) client_negotiation.cc:769] Sending connection context
0807 22:56:20.865335 (+ 101us) client_negotiation.cc:240] Negotiation successful
0807 22:56:20.865479 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":350,"threads_started":1}
I20260807 22:56:20.876329 28987 raft_consensus.cc:1261] T 24775f48f02549dab3e902d0663dc6a2 P 856a406170f44f378bfd5638cf75ee7a [term 1 FOLLOWER]: Refusing update from remote peer bb456655a12a41039f76160cfda886ea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 22:56:20.876331 29193 raft_consensus.cc:1261] T 24775f48f02549dab3e902d0663dc6a2 P 9e5fefbbe4064049919e70cf86eb9757 [term 1 FOLLOWER]: Refusing update from remote peer bb456655a12a41039f76160cfda886ea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 22:56:20.877065 29516 consensus_queue.cc:1035] T 24775f48f02549dab3e902d0663dc6a2 P bb456655a12a41039f76160cfda886ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e5fefbbe4064049919e70cf86eb9757" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 45043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 22:56:20.877543 29510 consensus_queue.cc:1035] T 24775f48f02549dab3e902d0663dc6a2 P bb456655a12a41039f76160cfda886ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "856a406170f44f378bfd5638cf75ee7a" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 22:56:20.913628 29501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:20.907513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.2:52671 (local address 127.25.94.3:45619)
0807 22:56:20.907697 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:20.907702 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:20.907728 (+ 26us) server_negotiation.cc:407] Connection header received
0807 22:56:20.907816 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:20.907821 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:20.907901 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:20.908000 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 22:56:20.908635 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:20.909615 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:20.911780 (+ 2165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:20.912239 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:20.912384 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:20.912833 (+ 449us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 22:56:20.912941 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 22:56:20.913040 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:20.913216 (+ 176us) server_negotiation.cc:299] Negotiation successful
0807 22:56:20.913296 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260807 22:56:20.913843 29563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:20.906721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.3:45619 (local address 127.25.94.2:52671)
0807 22:56:20.907428 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0807 22:56:20.907450 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 22:56:20.907585 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:56:20.908025 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:56:20.908034 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:56:20.908047 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0807 22:56:20.908467 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:20.908478 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:20.909793 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:20.909798 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:20.911623 (+ 1825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:20.911636 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:20.912433 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:20.912441 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:20.912584 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:20.912646 (+ 62us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0807 22:56:20.913043 (+ 397us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0807 22:56:20.913057 (+ 14us) client_negotiation.cc:769] Sending connection context
0807 22:56:20.913194 (+ 137us) client_negotiation.cc:240] Negotiation successful
0807 22:56:20.913337 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":1376,"threads_started":1}
I20260807 22:56:20.967190 28727 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42576:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 22:56:20.967525 28727 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42576:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 22:56:20.971753 28727 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1c2a3363ee8640eeb7d932cc0d4c104b: Sending DeleteTablet for 3 replicas of tablet 24775f48f02549dab3e902d0663dc6a2
I20260807 22:56:20.973572 28950 tablet_service.cc:1514] Processing DeleteTablet for tablet 24775f48f02549dab3e902d0663dc6a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:56:20 UTC) from {username='kudu', principal='kudu/127.25.94.62@KRBTEST.COM'} at 127.0.0.1:51654
I20260807 22:56:20.974424 29569 tablet_replica.cc:330] T 24775f48f02549dab3e902d0663dc6a2 P 856a406170f44f378bfd5638cf75ee7a: stopping tablet replica
I20260807 22:56:20.974362 28725 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42576:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260807 22:56:20.974871 29569 raft_consensus.cc:2229] T 24775f48f02549dab3e902d0663dc6a2 P 856a406170f44f378bfd5638cf75ee7a [term 1 FOLLOWER]: Raft consensus shutting down.
W20260807 22:56:20.975093 28725 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 22:56:20.975221 29569 raft_consensus.cc:2258] T 24775f48f02549dab3e902d0663dc6a2 P 856a406170f44f378bfd5638cf75ee7a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 22:56:20.976666 29163 tablet_service.cc:1514] Processing DeleteTablet for tablet 24775f48f02549dab3e902d0663dc6a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:56:20 UTC) from {username='kudu', principal='kudu/127.25.94.62@KRBTEST.COM'} at 127.0.0.1:55172
I20260807 22:56:20.981591 29362 tablet_service.cc:1514] Processing DeleteTablet for tablet 24775f48f02549dab3e902d0663dc6a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:56:20 UTC) from {username='kudu', principal='kudu/127.25.94.62@KRBTEST.COM'} at 127.0.0.1:59690
I20260807 22:56:20.983985 29569 ts_tablet_manager.cc:1905] T 24775f48f02549dab3e902d0663dc6a2 P 856a406170f44f378bfd5638cf75ee7a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 22:56:20.984432 28950 tablet_service.cc:1467] Processing CreateTablet for tablet 1fe8e977c836440cb9dc2a17391eed1a (DEFAULT_TABLE table=test-table [id=bfcd377ff8c14d549c36bacf7560670a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:56:20.984917 28950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fe8e977c836440cb9dc2a17391eed1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:56:20.990335 29161 tablet_service.cc:1467] Processing CreateTablet for tablet 1fe8e977c836440cb9dc2a17391eed1a (DEFAULT_TABLE table=test-table [id=bfcd377ff8c14d549c36bacf7560670a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:56:20.990924 29161 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fe8e977c836440cb9dc2a17391eed1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:56:20.992473 29360 tablet_service.cc:1467] Processing CreateTablet for tablet 1fe8e977c836440cb9dc2a17391eed1a (DEFAULT_TABLE table=test-table [id=bfcd377ff8c14d549c36bacf7560670a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:56:20.993033 29360 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fe8e977c836440cb9dc2a17391eed1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:56:21.001204 29504 tablet_bootstrap.cc:492] T 1fe8e977c836440cb9dc2a17391eed1a P bb456655a12a41039f76160cfda886ea: Bootstrap starting.
I20260807 22:56:21.003108 29504 tablet_bootstrap.cc:654] T 1fe8e977c836440cb9dc2a17391eed1a P bb456655a12a41039f76160cfda886ea: Neither blocks nor log segments found. Creating new log.
I20260807 22:56:21.012111 29503 tablet_bootstrap.cc:492] T 1fe8e977c836440cb9dc2a17391eed1a P 856a406170f44f378bfd5638cf75ee7a: Bootstrap starting.
I20260807 22:56:21.014194 29503 tablet_bootstrap.cc:654] T 1fe8e977c836440cb9dc2a17391eed1a P 856a406170f44f378bfd5638cf75ee7a: Neither blocks nor log segments found. Creating new log.
I20260807 22:56:21.018661 29569 ts_tablet_manager.cc:1918] T 24775f48f02549dab3e902d0663dc6a2 P 856a406170f44f378bfd5638cf75ee7a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 22:56:21.018853 29569 log.cc:1196] T 24775f48f02549dab3e902d0663dc6a2 P 856a406170f44f378bfd5638cf75ee7a: Deleting WAL directory at /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786143361779964-25976-0/minicluster-data/ts-0/wal/wals/24775f48f02549dab3e902d0663dc6a2
I20260807 22:56:21.019328 29569 ts_tablet_manager.cc:1939] T 24775f48f02549dab3e902d0663dc6a2 P 856a406170f44f378bfd5638cf75ee7a: Deleting consensus metadata
I20260807 22:56:21.021072 28698 catalog_manager.cc:4841] TS 856a406170f44f378bfd5638cf75ee7a (127.25.94.1:44541): tablet 24775f48f02549dab3e902d0663dc6a2 (table test-table [id=74072c6c61cf421abf36318aee46878a]) successfully deleted
I20260807 22:56:21.022328 29504 tablet_bootstrap.cc:492] T 1fe8e977c836440cb9dc2a17391eed1a P bb456655a12a41039f76160cfda886ea: No bootstrap required, opened a new log
I20260807 22:56:21.022490 29504 ts_tablet_manager.cc:1397] T 1fe8e977c836440cb9dc2a17391eed1a P bb456655a12a41039f76160cfda886ea: Time spent bootstrapping tablet: real 0.021s user 0.000s sys 0.009s
I20260807 22:56:21.023829 29502 tablet_bootstrap.cc:492] T 1fe8e977c836440cb9dc2a17391eed1a P 9e5fefbbe4064049919e70cf86eb9757: Bootstrap starting.
I20260807 22:56:21.024626 29504 raft_consensus.cc:348] T 1fe8e977c836440cb9dc2a17391eed1a P bb456655a12a41039f76160cfda886ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "856a406170f44f378bfd5638cf75ee7a" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44541 } } peers { permanent_uuid: "9e5fefbbe4064049919e70cf86eb9757" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 45043 } } peers { permanent_uuid: "bb456655a12a41039f76160cfda886ea" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 45619 } }
I20260807 22:56:21.024894 29504 raft_consensus.cc:374] T 1fe8e977c836440cb9dc2a17391eed1a P bb456655a12a41039f76160cfda886ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:56:21.024973 29504 raft_consensus.cc:729] T 1fe8e977c836440cb9dc2a17391eed1a P bb456655a12a41039f76160cfda886ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb456655a12a41039f76160cfda886ea, State: Initialized, Role: FOLLOWER
I20260807 22:56:21.025277 29504 consensus_queue.cc:260] T 1fe8e977c836440cb9dc2a17391eed1a P bb456655a12a41039f76160cfda886ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "856a406170f44f378bfd5638cf75ee7a" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44541 } } peers { permanent_uuid: "9e5fefbbe4064049919e70cf86eb9757" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 45043 } } peers { permanent_uuid: "bb456655a12a41039f76160cfda886ea" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 45619 } }
I20260807 22:56:21.025715 29502 tablet_bootstrap.cc:654] T 1fe8e977c836440cb9dc2a17391eed1a P 9e5fefbbe4064049919e70cf86eb9757: Neither blocks nor log segments found. Creating new log.
I20260807 22:56:21.025944 29504 ts_tablet_manager.cc:1428] T 1fe8e977c836440cb9dc2a17391eed1a P bb456655a12a41039f76160cfda886ea: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260807 22:56:21.039811 29503 tablet_bootstrap.cc:492] T 1fe8e977c836440cb9dc2a17391eed1a P 856a406170f44f378bfd5638cf75ee7a: No bootstrap required, opened a new log
I20260807 22:56:21.041278 29572 tablet_replica.cc:330] T 24775f48f02549dab3e902d0663dc6a2 P bb456655a12a41039f76160cfda886ea: stopping tablet replica
I20260807 22:56:21.041530 29502 tablet_bootstrap.cc:492] T 1fe8e977c836440cb9dc2a17391eed1a P 9e5fefbbe4064049919e70cf86eb9757: No bootstrap required, opened a new log
I20260807 22:56:21.041787 29502 ts_tablet_manager.cc:1397] T 1fe8e977c836440cb9dc2a17391eed1a P 9e5fefbbe4064049919e70cf86eb9757: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.008s
I20260807 22:56:21.041782 29572 raft_consensus.cc:2229] T 24775f48f02549dab3e902d0663dc6a2 P bb456655a12a41039f76160cfda886ea [term 1 LEADER]: Raft consensus shutting down.
I20260807 22:56:21.042821 29502 raft_consensus.cc:348] T 1fe8e977c836440cb9dc2a17391eed1a P 9e5fefbbe4064049919e70cf86eb9757 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "856a406170f44f378bfd5638cf75ee7a" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44541 } } peers { permanent_uuid: "9e5fefbbe4064049919e70cf86eb9757" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 45043 } } peers { permanent_uuid: "bb456655a12a41039f76160cfda886ea" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 45619 } }
I20260807 22:56:21.043196 29502 raft_consensus.cc:374] T 1fe8e977c836440cb9dc2a17391eed1a P 9e5fefbbe4064049919e70cf86eb9757 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:56:21.043388 29502 raft_consensus.cc:729] T 1fe8e977c836440cb9dc2a17391eed1a P 9e5fefbbe4064049919e70cf86eb9757 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e5fefbbe4064049919e70cf86eb9757, State: Initialized, Role: FOLLOWER
I20260807 22:56:21.043762 29502 consensus_queue.cc:260] T 1fe8e977c836440cb9dc2a17391eed1a P 9e5fefbbe4064049919e70cf86eb9757 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "856a406170f44f378bfd5638cf75ee7a" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44541 } } peers { permanent_uuid: "9e5fefbbe4064049919e70cf86eb9757" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 45043 } } peers { permanent_uuid: "bb456655a12a41039f76160cfda886ea" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 45619 } }
I20260807 22:56:21.043942 29572 raft_consensus.cc:2258] T 24775f48f02549dab3e902d0663dc6a2 P bb456655a12a41039f76160cfda886ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 22:56:21.045820 29502 ts_tablet_manager.cc:1428] T 1fe8e977c836440cb9dc2a17391eed1a P 9e5fefbbe4064049919e70cf86eb9757: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260807 22:56:21.046092 29503 ts_tablet_manager.cc:1397] T 1fe8e977c836440cb9dc2a17391eed1a P 856a406170f44f378bfd5638cf75ee7a: Time spent bootstrapping tablet: real 0.034s user 0.004s sys 0.009s
I20260807 22:56:21.046938 29503 raft_consensus.cc:348] T 1fe8e977c836440cb9dc2a17391eed1a P 856a406170f44f378bfd5638cf75ee7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "856a406170f44f378bfd5638cf75ee7a" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44541 } } peers { permanent_uuid: "9e5fefbbe4064049919e70cf86eb9757" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 45043 } } peers { permanent_uuid: "bb456655a12a41039f76160cfda886ea" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 45619 } }
I20260807 22:56:21.047240 29503 raft_consensus.cc:374] T 1fe8e977c836440cb9dc2a17391eed1a P 856a406170f44f378bfd5638cf75ee7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:56:21.047355 29503 raft_consensus.cc:729] T 1fe8e977c836440cb9dc2a17391eed1a P 856a406170f44f378bfd5638cf75ee7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 856a406170f44f378bfd5638cf75ee7a, State: Initialized, Role: FOLLOWER
I20260807 22:56:21.047633 29503 consensus_queue.cc:260] T 1fe8e977c836440cb9dc2a17391eed1a P 856a406170f44f378bfd5638cf75ee7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "856a406170f44f378bfd5638cf75ee7a" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44541 } } peers { permanent_uuid: "9e5fefbbe4064049919e70cf86eb9757" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 45043 } } peers { permanent_uuid: "bb456655a12a41039f76160cfda886ea" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 45619 } }
I20260807 22:56:21.048187 29503 ts_tablet_manager.cc:1428] T 1fe8e977c836440cb9dc2a17391eed1a P 856a406170f44f378bfd5638cf75ee7a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260807 22:56:21.049930 29572 ts_tablet_manager.cc:1905] T 24775f48f02549dab3e902d0663dc6a2 P bb456655a12a41039f76160cfda886ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 22:56:21.056375 29570 tablet_replica.cc:330] T 24775f48f02549dab3e902d0663dc6a2 P 9e5fefbbe4064049919e70cf86eb9757: stopping tablet replica
I20260807 22:56:21.056941 29570 raft_consensus.cc:2229] T 24775f48f02549dab3e902d0663dc6a2 P 9e5fefbbe4064049919e70cf86eb9757 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 22:56:21.057420 29570 raft_consensus.cc:2258] T 24775f48f02549dab3e902d0663dc6a2 P 9e5fefbbe4064049919e70cf86eb9757 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 22:56:21.058535 29572 ts_tablet_manager.cc:1918] T 24775f48f02549dab3e902d0663dc6a2 P bb456655a12a41039f76160cfda886ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 22:56:21.058758 29572 log.cc:1196] T 24775f48f02549dab3e902d0663dc6a2 P bb456655a12a41039f76160cfda886ea: Deleting WAL directory at /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786143361779964-25976-0/minicluster-data/ts-2/wal/wals/24775f48f02549dab3e902d0663dc6a2
I20260807 22:56:21.059362 29572 ts_tablet_manager.cc:1939] T 24775f48f02549dab3e902d0663dc6a2 P bb456655a12a41039f76160cfda886ea: Deleting consensus metadata
I20260807 22:56:21.059824 29570 ts_tablet_manager.cc:1905] T 24775f48f02549dab3e902d0663dc6a2 P 9e5fefbbe4064049919e70cf86eb9757: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 22:56:21.061281 28700 catalog_manager.cc:4841] TS bb456655a12a41039f76160cfda886ea (127.25.94.3:45619): tablet 24775f48f02549dab3e902d0663dc6a2 (table test-table [id=74072c6c61cf421abf36318aee46878a]) successfully deleted
I20260807 22:56:21.062929 29512 raft_consensus.cc:482] T 1fe8e977c836440cb9dc2a17391eed1a P 856a406170f44f378bfd5638cf75ee7a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 22:56:21.063100 29512 raft_consensus.cc:504] T 1fe8e977c836440cb9dc2a17391eed1a P 856a406170f44f378bfd5638cf75ee7a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "856a406170f44f378bfd5638cf75ee7a" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44541 } } peers { permanent_uuid: "9e5fefbbe4064049919e70cf86eb9757" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 45043 } } peers { permanent_uuid: "bb456655a12a41039f76160cfda886ea" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 45619 } }
I20260807 22:56:21.063778 29512 leader_election.cc:288] T 1fe8e977c836440cb9dc2a17391eed1a P 856a406170f44f378bfd5638cf75ee7a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9e5fefbbe4064049919e70cf86eb9757 (127.25.94.2:45043), bb456655a12a41039f76160cfda886ea (127.25.94.3:45619)
I20260807 22:56:21.064913 29192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fe8e977c836440cb9dc2a17391eed1a" candidate_uuid: "856a406170f44f378bfd5638cf75ee7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e5fefbbe4064049919e70cf86eb9757" is_pre_election: true
I20260807 22:56:21.065305 29192 raft_consensus.cc:2454] T 1fe8e977c836440cb9dc2a17391eed1a P 9e5fefbbe4064049919e70cf86eb9757 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 856a406170f44f378bfd5638cf75ee7a in term 0.
I20260807 22:56:21.065732 29407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fe8e977c836440cb9dc2a17391eed1a" candidate_uuid: "856a406170f44f378bfd5638cf75ee7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb456655a12a41039f76160cfda886ea" is_pre_election: true
I20260807 22:56:21.066023 29407 raft_consensus.cc:2454] T 1fe8e977c836440cb9dc2a17391eed1a P bb456655a12a41039f76160cfda886ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 856a406170f44f378bfd5638cf75ee7a in term 0.
I20260807 22:56:21.066661 28871 leader_election.cc:302] T 1fe8e977c836440cb9dc2a17391eed1a P 856a406170f44f378bfd5638cf75ee7a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 856a406170f44f378bfd5638cf75ee7a, bb456655a12a41039f76160cfda886ea; no voters:
I20260807 22:56:21.067441 29512 raft_consensus.cc:2781] T 1fe8e977c836440cb9dc2a17391eed1a P 856a406170f44f378bfd5638cf75ee7a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 22:56:21.067588 29512 raft_consensus.cc:482] T 1fe8e977c836440cb9dc2a17391eed1a P 856a406170f44f378bfd5638cf75ee7a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 22:56:21.067698 29512 raft_consensus.cc:3037] T 1fe8e977c836440cb9dc2a17391eed1a P 856a406170f44f378bfd5638cf75ee7a [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:56:21.069288 29570 ts_tablet_manager.cc:1918] T 24775f48f02549dab3e902d0663dc6a2 P 9e5fefbbe4064049919e70cf86eb9757: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 22:56:21.069559 29570 log.cc:1196] T 24775f48f02549dab3e902d0663dc6a2 P 9e5fefbbe4064049919e70cf86eb9757: Deleting WAL directory at /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786143361779964-25976-0/minicluster-data/ts-1/wal/wals/24775f48f02549dab3e902d0663dc6a2
I20260807 22:56:21.070546 29570 ts_tablet_manager.cc:1939] T 24775f48f02549dab3e902d0663dc6a2 P 9e5fefbbe4064049919e70cf86eb9757: Deleting consensus metadata
I20260807 22:56:21.072947 28700 catalog_manager.cc:4841] TS 9e5fefbbe4064049919e70cf86eb9757 (127.25.94.2:45043): tablet 24775f48f02549dab3e902d0663dc6a2 (table test-table [id=74072c6c61cf421abf36318aee46878a]) successfully deleted
I20260807 22:56:21.073230 29512 raft_consensus.cc:504] T 1fe8e977c836440cb9dc2a17391eed1a P 856a406170f44f378bfd5638cf75ee7a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "856a406170f44f378bfd5638cf75ee7a" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44541 } } peers { permanent_uuid: "9e5fefbbe4064049919e70cf86eb9757" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 45043 } } peers { permanent_uuid: "bb456655a12a41039f76160cfda886ea" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 45619 } }
I20260807 22:56:21.073920 29512 leader_election.cc:288] T 1fe8e977c836440cb9dc2a17391eed1a P 856a406170f44f378bfd5638cf75ee7a [CANDIDATE]: Term 1 election: Requested vote from peers 9e5fefbbe4064049919e70cf86eb9757 (127.25.94.2:45043), bb456655a12a41039f76160cfda886ea (127.25.94.3:45619)
I20260807 22:56:21.074522 29407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fe8e977c836440cb9dc2a17391eed1a" candidate_uuid: "856a406170f44f378bfd5638cf75ee7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb456655a12a41039f76160cfda886ea"
I20260807 22:56:21.074812 29407 raft_consensus.cc:3037] T 1fe8e977c836440cb9dc2a17391eed1a P bb456655a12a41039f76160cfda886ea [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:56:21.074816 29192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fe8e977c836440cb9dc2a17391eed1a" candidate_uuid: "856a406170f44f378bfd5638cf75ee7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e5fefbbe4064049919e70cf86eb9757"
I20260807 22:56:21.075045 29192 raft_consensus.cc:3037] T 1fe8e977c836440cb9dc2a17391eed1a P 9e5fefbbe4064049919e70cf86eb9757 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:56:21.077425 29407 raft_consensus.cc:2454] T 1fe8e977c836440cb9dc2a17391eed1a P bb456655a12a41039f76160cfda886ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 856a406170f44f378bfd5638cf75ee7a in term 1.
I20260807 22:56:21.077940 28871 leader_election.cc:302] T 1fe8e977c836440cb9dc2a17391eed1a P 856a406170f44f378bfd5638cf75ee7a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 856a406170f44f378bfd5638cf75ee7a, bb456655a12a41039f76160cfda886ea; no voters:
I20260807 22:56:21.078372 29512 raft_consensus.cc:2781] T 1fe8e977c836440cb9dc2a17391eed1a P 856a406170f44f378bfd5638cf75ee7a [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:56:21.078506 29192 raft_consensus.cc:2454] T 1fe8e977c836440cb9dc2a17391eed1a P 9e5fefbbe4064049919e70cf86eb9757 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 856a406170f44f378bfd5638cf75ee7a in term 1.
I20260807 22:56:21.079648 29512 raft_consensus.cc:686] T 1fe8e977c836440cb9dc2a17391eed1a P 856a406170f44f378bfd5638cf75ee7a [term 1 LEADER]: Becoming Leader. State: Replica: 856a406170f44f378bfd5638cf75ee7a, State: Running, Role: LEADER
I20260807 22:56:21.080188 29512 consensus_queue.cc:237] T 1fe8e977c836440cb9dc2a17391eed1a P 856a406170f44f378bfd5638cf75ee7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "856a406170f44f378bfd5638cf75ee7a" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44541 } } peers { permanent_uuid: "9e5fefbbe4064049919e70cf86eb9757" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 45043 } } peers { permanent_uuid: "bb456655a12a41039f76160cfda886ea" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 45619 } }
I20260807 22:56:21.084354 28725 catalog_manager.cc:5495] T 1fe8e977c836440cb9dc2a17391eed1a P 856a406170f44f378bfd5638cf75ee7a reported cstate change: term changed from 0 to 1, leader changed from <none> to 856a406170f44f378bfd5638cf75ee7a (127.25.94.1). New cstate: current_term: 1 leader_uuid: "856a406170f44f378bfd5638cf75ee7a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "856a406170f44f378bfd5638cf75ee7a" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44541 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e5fefbbe4064049919e70cf86eb9757" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 45043 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb456655a12a41039f76160cfda886ea" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 45619 } health_report { overall_health: UNKNOWN } } }
I20260807 22:56:21.117865 29498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:21.110151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51662 (local address 127.25.94.1:44541)
0807 22:56:21.110386 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:21.110393 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:21.110626 (+ 233us) server_negotiation.cc:407] Connection header received
0807 22:56:21.110924 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:21.110931 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:21.111044 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:21.111237 (+ 193us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 22:56:21.111693 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:21.112778 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:21.114888 (+ 2110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:21.115547 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:21.115692 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:21.116261 (+ 569us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 22:56:21.117092 (+ 831us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 22:56:21.117381 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:21.117448 (+ 67us) server_negotiation.cc:299] Negotiation successful
0807 22:56:21.117546 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260807 22:56:21.146580 29494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:21.130841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.1:38549 (local address 127.25.94.2:45043)
0807 22:56:21.131049 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:21.131056 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:21.139566 (+ 8510us) server_negotiation.cc:407] Connection header received
0807 22:56:21.139821 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:21.139828 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:21.139930 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:21.140045 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 22:56:21.140686 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:21.141969 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:21.144100 (+ 2131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:21.144859 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:21.145000 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:21.145668 (+ 668us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 22:56:21.145846 (+ 178us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 22:56:21.145950 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:21.146139 (+ 189us) server_negotiation.cc:299] Negotiation successful
0807 22:56:21.146235 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260807 22:56:21.146721 29587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:21.130703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.2:45043 (local address 127.25.94.1:38549)
0807 22:56:21.139386 (+ 8683us) negotiation.cc:107] Waiting for socket to connect
0807 22:56:21.139412 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0807 22:56:21.139648 (+ 236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:56:21.140097 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:56:21.140103 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:56:21.140118 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0807 22:56:21.140475 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:21.140489 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:21.142146 (+ 1657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:21.142153 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:21.143920 (+ 1767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:21.143938 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:21.145062 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:21.145069 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:21.145350 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:21.145442 (+ 92us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0807 22:56:21.145983 (+ 541us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0807 22:56:21.145998 (+ 15us) client_negotiation.cc:769] Sending connection context
0807 22:56:21.146099 (+ 101us) client_negotiation.cc:240] Negotiation successful
0807 22:56:21.146258 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8429,"thread_start_us":6695,"threads_started":1}
I20260807 22:56:21.158686 29192 raft_consensus.cc:1261] T 1fe8e977c836440cb9dc2a17391eed1a P 9e5fefbbe4064049919e70cf86eb9757 [term 1 FOLLOWER]: Refusing update from remote peer 856a406170f44f378bfd5638cf75ee7a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 22:56:21.159578 29576 consensus_queue.cc:1035] T 1fe8e977c836440cb9dc2a17391eed1a P 856a406170f44f378bfd5638cf75ee7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e5fefbbe4064049919e70cf86eb9757" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 45043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 22:56:21.162024 29407 raft_consensus.cc:1261] T 1fe8e977c836440cb9dc2a17391eed1a P bb456655a12a41039f76160cfda886ea [term 1 FOLLOWER]: Refusing update from remote peer 856a406170f44f378bfd5638cf75ee7a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 22:56:21.163280 29576 consensus_queue.cc:1035] T 1fe8e977c836440cb9dc2a17391eed1a P 856a406170f44f378bfd5638cf75ee7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb456655a12a41039f76160cfda886ea" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 45619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 22:56:21.172008 29542 mvcc.cc:205] Tried to move back new op lower bound from 7316043289217724416 to 7316043288909291520. Current Snapshot: MvccSnapshot[applied={T|T < 7316043289217724416}]
I20260807 22:56:21.211727 29498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:21.201746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.2:42203 (local address 127.25.94.1:44541)
0807 22:56:21.201910 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:21.201918 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:21.202529 (+ 611us) server_negotiation.cc:407] Connection header received
0807 22:56:21.202680 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:21.202686 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:21.202783 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:21.202879 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 22:56:21.203649 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:21.204795 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:21.206934 (+ 2139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:21.207754 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:21.207892 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:21.210768 (+ 2876us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 22:56:21.210944 (+ 176us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 22:56:21.211077 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:21.211303 (+ 226us) server_negotiation.cc:299] Negotiation successful
0807 22:56:21.211405 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260807 22:56:21.211795 29563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:21.202134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.1:44541 (local address 127.25.94.2:42203)
0807 22:56:21.202386 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0807 22:56:21.202412 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0807 22:56:21.202555 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:56:21.203053 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:56:21.203059 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:56:21.203073 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0807 22:56:21.203407 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:21.203426 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:21.204974 (+ 1548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:21.204998 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:21.206767 (+ 1769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:21.206784 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:21.210122 (+ 3338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:21.210129 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:21.210473 (+ 344us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:21.210540 (+ 67us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0807 22:56:21.211136 (+ 596us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0807 22:56:21.211149 (+ 13us) client_negotiation.cc:769] Sending connection context
0807 22:56:21.211243 (+ 94us) client_negotiation.cc:240] Negotiation successful
0807 22:56:21.211446 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78}
W20260807 22:56:21.213570 29542 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260807 22:56:22.861364 28727 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42576:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 22:56:22.861636 28727 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42576:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 22:56:22.866390 28727 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1c2a3363ee8640eeb7d932cc0d4c104b: Sending DeleteTablet for 3 replicas of tablet 1fe8e977c836440cb9dc2a17391eed1a
I20260807 22:56:22.867280 28950 tablet_service.cc:1514] Processing DeleteTablet for tablet 1fe8e977c836440cb9dc2a17391eed1a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:56:22 UTC) from {username='kudu', principal='kudu/127.25.94.62@KRBTEST.COM'} at 127.0.0.1:51654
I20260807 22:56:22.867395 29362 tablet_service.cc:1514] Processing DeleteTablet for tablet 1fe8e977c836440cb9dc2a17391eed1a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:56:22 UTC) from {username='kudu', principal='kudu/127.25.94.62@KRBTEST.COM'} at 127.0.0.1:59690
I20260807 22:56:22.867347 29163 tablet_service.cc:1514] Processing DeleteTablet for tablet 1fe8e977c836440cb9dc2a17391eed1a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:56:22 UTC) from {username='kudu', principal='kudu/127.25.94.62@KRBTEST.COM'} at 127.0.0.1:55172
I20260807 22:56:22.881410 29635 tablet_replica.cc:330] T 1fe8e977c836440cb9dc2a17391eed1a P 856a406170f44f378bfd5638cf75ee7a: stopping tablet replica
I20260807 22:56:22.881735 29635 raft_consensus.cc:2229] T 1fe8e977c836440cb9dc2a17391eed1a P 856a406170f44f378bfd5638cf75ee7a [term 1 LEADER]: Raft consensus shutting down.
I20260807 22:56:22.882362 29638 tablet_replica.cc:330] T 1fe8e977c836440cb9dc2a17391eed1a P bb456655a12a41039f76160cfda886ea: stopping tablet replica
I20260807 22:56:22.882421 29635 raft_consensus.cc:2258] T 1fe8e977c836440cb9dc2a17391eed1a P 856a406170f44f378bfd5638cf75ee7a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 22:56:22.882587 29636 tablet_replica.cc:330] T 1fe8e977c836440cb9dc2a17391eed1a P 9e5fefbbe4064049919e70cf86eb9757: stopping tablet replica
I20260807 22:56:22.882725 29638 raft_consensus.cc:2229] T 1fe8e977c836440cb9dc2a17391eed1a P bb456655a12a41039f76160cfda886ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 22:56:22.882799 29636 raft_consensus.cc:2229] T 1fe8e977c836440cb9dc2a17391eed1a P 9e5fefbbe4064049919e70cf86eb9757 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 22:56:22.883049 29636 raft_consensus.cc:2258] T 1fe8e977c836440cb9dc2a17391eed1a P 9e5fefbbe4064049919e70cf86eb9757 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 22:56:22.883179 29638 raft_consensus.cc:2258] T 1fe8e977c836440cb9dc2a17391eed1a P bb456655a12a41039f76160cfda886ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 22:56:22.884171 29636 ts_tablet_manager.cc:1905] T 1fe8e977c836440cb9dc2a17391eed1a P 9e5fefbbe4064049919e70cf86eb9757: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 22:56:22.884248 29638 ts_tablet_manager.cc:1905] T 1fe8e977c836440cb9dc2a17391eed1a P bb456655a12a41039f76160cfda886ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 22:56:22.884698 29635 ts_tablet_manager.cc:1905] T 1fe8e977c836440cb9dc2a17391eed1a P 856a406170f44f378bfd5638cf75ee7a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 22:56:22.891620 29638 ts_tablet_manager.cc:1918] T 1fe8e977c836440cb9dc2a17391eed1a P bb456655a12a41039f76160cfda886ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260807 22:56:22.891865 29638 log.cc:1196] T 1fe8e977c836440cb9dc2a17391eed1a P bb456655a12a41039f76160cfda886ea: Deleting WAL directory at /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786143361779964-25976-0/minicluster-data/ts-2/wal/wals/1fe8e977c836440cb9dc2a17391eed1a
I20260807 22:56:22.892074 29635 ts_tablet_manager.cc:1918] T 1fe8e977c836440cb9dc2a17391eed1a P 856a406170f44f378bfd5638cf75ee7a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260807 22:56:22.892254 29635 log.cc:1196] T 1fe8e977c836440cb9dc2a17391eed1a P 856a406170f44f378bfd5638cf75ee7a: Deleting WAL directory at /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786143361779964-25976-0/minicluster-data/ts-0/wal/wals/1fe8e977c836440cb9dc2a17391eed1a
I20260807 22:56:22.892566 29638 ts_tablet_manager.cc:1939] T 1fe8e977c836440cb9dc2a17391eed1a P bb456655a12a41039f76160cfda886ea: Deleting consensus metadata
I20260807 22:56:22.892671 29636 ts_tablet_manager.cc:1918] T 1fe8e977c836440cb9dc2a17391eed1a P 9e5fefbbe4064049919e70cf86eb9757: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260807 22:56:22.892822 29635 ts_tablet_manager.cc:1939] T 1fe8e977c836440cb9dc2a17391eed1a P 856a406170f44f378bfd5638cf75ee7a: Deleting consensus metadata
I20260807 22:56:22.892894 29636 log.cc:1196] T 1fe8e977c836440cb9dc2a17391eed1a P 9e5fefbbe4064049919e70cf86eb9757: Deleting WAL directory at /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786143361779964-25976-0/minicluster-data/ts-1/wal/wals/1fe8e977c836440cb9dc2a17391eed1a
I20260807 22:56:22.893520 29636 ts_tablet_manager.cc:1939] T 1fe8e977c836440cb9dc2a17391eed1a P 9e5fefbbe4064049919e70cf86eb9757: Deleting consensus metadata
I20260807 22:56:22.893970 28700 catalog_manager.cc:4841] TS bb456655a12a41039f76160cfda886ea (127.25.94.3:45619): tablet 1fe8e977c836440cb9dc2a17391eed1a (table test-table [id=bfcd377ff8c14d549c36bacf7560670a]) successfully deleted
I20260807 22:56:22.894133 28698 catalog_manager.cc:4841] TS 856a406170f44f378bfd5638cf75ee7a (127.25.94.1:44541): tablet 1fe8e977c836440cb9dc2a17391eed1a (table test-table [id=bfcd377ff8c14d549c36bacf7560670a]) successfully deleted
I20260807 22:56:22.894771 28700 catalog_manager.cc:4841] TS 9e5fefbbe4064049919e70cf86eb9757 (127.25.94.2:45043): tablet 1fe8e977c836440cb9dc2a17391eed1a (table test-table [id=bfcd377ff8c14d549c36bacf7560670a]) successfully deleted
Aug 07 22:56:22 dist-test-slave-z8x9 krb5kdc[28646](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143382, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 07 22:56:22 dist-test-slave-z8x9 krb5kdc[28646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143382, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.25.94.62@KRBTEST.COM
I20260807 22:56:22.932551 29652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:22.918004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47734 (local address 127.25.94.62:34227)
0807 22:56:22.918610 (+ 606us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:22.918614 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:22.918637 (+ 23us) server_negotiation.cc:407] Connection header received
0807 22:56:22.918757 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:22.918761 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:22.918836 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:22.918927 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:56:22.920229 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:22.921004 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:22.922368 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:22.922740 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:22.922856 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:22.927300 (+ 4444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:56:22.927306 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:56:22.927309 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:56:22.927345 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:56:22.930166 (+ 2821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:22.930770 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:22.930773 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:22.930776 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:22.930900 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:22.931207 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:22.931210 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:22.931212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:22.931541 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:56:22.931791 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:22.932070 (+ 279us) server_negotiation.cc:299] Negotiation successful
0807 22:56:22.932250 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":493,"thread_start_us":334,"threads_started":1}
I20260807 22:56:22.936647 28727 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47734:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 22:56:22.937278 28727 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 22:56:22.944021 28950 tablet_service.cc:1467] Processing CreateTablet for tablet b1299a3340954c93b69e56dbccaca18e (DEFAULT_TABLE table=test-table [id=62992ca864d3442b929fad67b8276dff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:56:22.944021 29362 tablet_service.cc:1467] Processing CreateTablet for tablet b1299a3340954c93b69e56dbccaca18e (DEFAULT_TABLE table=test-table [id=62992ca864d3442b929fad67b8276dff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:56:22.944021 29163 tablet_service.cc:1467] Processing CreateTablet for tablet b1299a3340954c93b69e56dbccaca18e (DEFAULT_TABLE table=test-table [id=62992ca864d3442b929fad67b8276dff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:56:22.944481 29163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1299a3340954c93b69e56dbccaca18e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:56:22.944499 29362 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1299a3340954c93b69e56dbccaca18e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:56:22.944499 28950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1299a3340954c93b69e56dbccaca18e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:56:22.950691 29655 tablet_bootstrap.cc:492] T b1299a3340954c93b69e56dbccaca18e P 9e5fefbbe4064049919e70cf86eb9757: Bootstrap starting.
I20260807 22:56:22.951512 29656 tablet_bootstrap.cc:492] T b1299a3340954c93b69e56dbccaca18e P bb456655a12a41039f76160cfda886ea: Bootstrap starting.
I20260807 22:56:22.953629 29658 tablet_bootstrap.cc:492] T b1299a3340954c93b69e56dbccaca18e P 856a406170f44f378bfd5638cf75ee7a: Bootstrap starting.
I20260807 22:56:22.954320 29656 tablet_bootstrap.cc:654] T b1299a3340954c93b69e56dbccaca18e P bb456655a12a41039f76160cfda886ea: Neither blocks nor log segments found. Creating new log.
I20260807 22:56:22.955120 29655 tablet_bootstrap.cc:654] T b1299a3340954c93b69e56dbccaca18e P 9e5fefbbe4064049919e70cf86eb9757: Neither blocks nor log segments found. Creating new log.
I20260807 22:56:22.956295 29658 tablet_bootstrap.cc:654] T b1299a3340954c93b69e56dbccaca18e P 856a406170f44f378bfd5638cf75ee7a: Neither blocks nor log segments found. Creating new log.
I20260807 22:56:22.958148 29656 tablet_bootstrap.cc:492] T b1299a3340954c93b69e56dbccaca18e P bb456655a12a41039f76160cfda886ea: No bootstrap required, opened a new log
I20260807 22:56:22.958432 29656 ts_tablet_manager.cc:1397] T b1299a3340954c93b69e56dbccaca18e P bb456655a12a41039f76160cfda886ea: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.006s
I20260807 22:56:22.959532 29656 raft_consensus.cc:348] T b1299a3340954c93b69e56dbccaca18e P bb456655a12a41039f76160cfda886ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "856a406170f44f378bfd5638cf75ee7a" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44541 } } peers { permanent_uuid: "9e5fefbbe4064049919e70cf86eb9757" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 45043 } } peers { permanent_uuid: "bb456655a12a41039f76160cfda886ea" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 45619 } }
I20260807 22:56:22.959839 29656 raft_consensus.cc:374] T b1299a3340954c93b69e56dbccaca18e P bb456655a12a41039f76160cfda886ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:56:22.959937 29656 raft_consensus.cc:729] T b1299a3340954c93b69e56dbccaca18e P bb456655a12a41039f76160cfda886ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb456655a12a41039f76160cfda886ea, State: Initialized, Role: FOLLOWER
I20260807 22:56:22.960203 29656 consensus_queue.cc:260] T b1299a3340954c93b69e56dbccaca18e P bb456655a12a41039f76160cfda886ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "856a406170f44f378bfd5638cf75ee7a" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44541 } } peers { permanent_uuid: "9e5fefbbe4064049919e70cf86eb9757" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 45043 } } peers { permanent_uuid: "bb456655a12a41039f76160cfda886ea" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 45619 } }
I20260807 22:56:22.961158 29658 tablet_bootstrap.cc:492] T b1299a3340954c93b69e56dbccaca18e P 856a406170f44f378bfd5638cf75ee7a: No bootstrap required, opened a new log
I20260807 22:56:22.961400 29658 ts_tablet_manager.cc:1397] T b1299a3340954c93b69e56dbccaca18e P 856a406170f44f378bfd5638cf75ee7a: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.003s
I20260807 22:56:22.961659 29656 ts_tablet_manager.cc:1428] T b1299a3340954c93b69e56dbccaca18e P bb456655a12a41039f76160cfda886ea: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260807 22:56:22.962203 29658 raft_consensus.cc:348] T b1299a3340954c93b69e56dbccaca18e P 856a406170f44f378bfd5638cf75ee7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "856a406170f44f378bfd5638cf75ee7a" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44541 } } peers { permanent_uuid: "9e5fefbbe4064049919e70cf86eb9757" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 45043 } } peers { permanent_uuid: "bb456655a12a41039f76160cfda886ea" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 45619 } }
I20260807 22:56:22.962483 29658 raft_consensus.cc:374] T b1299a3340954c93b69e56dbccaca18e P 856a406170f44f378bfd5638cf75ee7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:56:22.962625 29655 tablet_bootstrap.cc:492] T b1299a3340954c93b69e56dbccaca18e P 9e5fefbbe4064049919e70cf86eb9757: No bootstrap required, opened a new log
I20260807 22:56:22.962634 29658 raft_consensus.cc:729] T b1299a3340954c93b69e56dbccaca18e P 856a406170f44f378bfd5638cf75ee7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 856a406170f44f378bfd5638cf75ee7a, State: Initialized, Role: FOLLOWER
I20260807 22:56:22.962939 29655 ts_tablet_manager.cc:1397] T b1299a3340954c93b69e56dbccaca18e P 9e5fefbbe4064049919e70cf86eb9757: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.006s
I20260807 22:56:22.962997 29658 consensus_queue.cc:260] T b1299a3340954c93b69e56dbccaca18e P 856a406170f44f378bfd5638cf75ee7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "856a406170f44f378bfd5638cf75ee7a" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44541 } } peers { permanent_uuid: "9e5fefbbe4064049919e70cf86eb9757" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 45043 } } peers { permanent_uuid: "bb456655a12a41039f76160cfda886ea" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 45619 } }
I20260807 22:56:22.963820 29658 ts_tablet_manager.cc:1428] T b1299a3340954c93b69e56dbccaca18e P 856a406170f44f378bfd5638cf75ee7a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260807 22:56:22.964555 29655 raft_consensus.cc:348] T b1299a3340954c93b69e56dbccaca18e P 9e5fefbbe4064049919e70cf86eb9757 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "856a406170f44f378bfd5638cf75ee7a" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44541 } } peers { permanent_uuid: "9e5fefbbe4064049919e70cf86eb9757" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 45043 } } peers { permanent_uuid: "bb456655a12a41039f76160cfda886ea" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 45619 } }
I20260807 22:56:22.964834 29655 raft_consensus.cc:374] T b1299a3340954c93b69e56dbccaca18e P 9e5fefbbe4064049919e70cf86eb9757 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:56:22.964958 29655 raft_consensus.cc:729] T b1299a3340954c93b69e56dbccaca18e P 9e5fefbbe4064049919e70cf86eb9757 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e5fefbbe4064049919e70cf86eb9757, State: Initialized, Role: FOLLOWER
I20260807 22:56:22.965317 29655 consensus_queue.cc:260] T b1299a3340954c93b69e56dbccaca18e P 9e5fefbbe4064049919e70cf86eb9757 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "856a406170f44f378bfd5638cf75ee7a" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44541 } } peers { permanent_uuid: "9e5fefbbe4064049919e70cf86eb9757" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 45043 } } peers { permanent_uuid: "bb456655a12a41039f76160cfda886ea" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 45619 } }
I20260807 22:56:22.966084 29655 ts_tablet_manager.cc:1428] T b1299a3340954c93b69e56dbccaca18e P 9e5fefbbe4064049919e70cf86eb9757: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260807 22:56:22.968688 29600 raft_consensus.cc:482] T b1299a3340954c93b69e56dbccaca18e P 9e5fefbbe4064049919e70cf86eb9757 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 22:56:22.968864 29600 raft_consensus.cc:504] T b1299a3340954c93b69e56dbccaca18e P 9e5fefbbe4064049919e70cf86eb9757 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "856a406170f44f378bfd5638cf75ee7a" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44541 } } peers { permanent_uuid: "9e5fefbbe4064049919e70cf86eb9757" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 45043 } } peers { permanent_uuid: "bb456655a12a41039f76160cfda886ea" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 45619 } }
I20260807 22:56:22.969554 29600 leader_election.cc:288] T b1299a3340954c93b69e56dbccaca18e P 9e5fefbbe4064049919e70cf86eb9757 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 856a406170f44f378bfd5638cf75ee7a (127.25.94.1:44541), bb456655a12a41039f76160cfda886ea (127.25.94.3:45619)
I20260807 22:56:22.970037 28985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1299a3340954c93b69e56dbccaca18e" candidate_uuid: "9e5fefbbe4064049919e70cf86eb9757" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "856a406170f44f378bfd5638cf75ee7a" is_pre_election: true
I20260807 22:56:22.970407 28985 raft_consensus.cc:2454] T b1299a3340954c93b69e56dbccaca18e P 856a406170f44f378bfd5638cf75ee7a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e5fefbbe4064049919e70cf86eb9757 in term 0.
I20260807 22:56:22.970340 29406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1299a3340954c93b69e56dbccaca18e" candidate_uuid: "9e5fefbbe4064049919e70cf86eb9757" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb456655a12a41039f76160cfda886ea" is_pre_election: true
I20260807 22:56:22.970598 29406 raft_consensus.cc:2454] T b1299a3340954c93b69e56dbccaca18e P bb456655a12a41039f76160cfda886ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e5fefbbe4064049919e70cf86eb9757 in term 0.
I20260807 22:56:22.970908 29094 leader_election.cc:302] T b1299a3340954c93b69e56dbccaca18e P 9e5fefbbe4064049919e70cf86eb9757 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 856a406170f44f378bfd5638cf75ee7a, 9e5fefbbe4064049919e70cf86eb9757; no voters:
I20260807 22:56:22.971220 29600 raft_consensus.cc:2781] T b1299a3340954c93b69e56dbccaca18e P 9e5fefbbe4064049919e70cf86eb9757 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 22:56:22.971321 29600 raft_consensus.cc:482] T b1299a3340954c93b69e56dbccaca18e P 9e5fefbbe4064049919e70cf86eb9757 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 22:56:22.971432 29600 raft_consensus.cc:3037] T b1299a3340954c93b69e56dbccaca18e P 9e5fefbbe4064049919e70cf86eb9757 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:56:22.973659 29600 raft_consensus.cc:504] T b1299a3340954c93b69e56dbccaca18e P 9e5fefbbe4064049919e70cf86eb9757 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "856a406170f44f378bfd5638cf75ee7a" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44541 } } peers { permanent_uuid: "9e5fefbbe4064049919e70cf86eb9757" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 45043 } } peers { permanent_uuid: "bb456655a12a41039f76160cfda886ea" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 45619 } }
I20260807 22:56:22.974403 29600 leader_election.cc:288] T b1299a3340954c93b69e56dbccaca18e P 9e5fefbbe4064049919e70cf86eb9757 [CANDIDATE]: Term 1 election: Requested vote from peers 856a406170f44f378bfd5638cf75ee7a (127.25.94.1:44541), bb456655a12a41039f76160cfda886ea (127.25.94.3:45619)
I20260807 22:56:22.974810 28985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1299a3340954c93b69e56dbccaca18e" candidate_uuid: "9e5fefbbe4064049919e70cf86eb9757" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "856a406170f44f378bfd5638cf75ee7a"
I20260807 22:56:22.975073 28985 raft_consensus.cc:3037] T b1299a3340954c93b69e56dbccaca18e P 856a406170f44f378bfd5638cf75ee7a [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:56:22.975008 29406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1299a3340954c93b69e56dbccaca18e" candidate_uuid: "9e5fefbbe4064049919e70cf86eb9757" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb456655a12a41039f76160cfda886ea"
I20260807 22:56:22.975265 29406 raft_consensus.cc:3037] T b1299a3340954c93b69e56dbccaca18e P bb456655a12a41039f76160cfda886ea [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:56:22.978480 29406 raft_consensus.cc:2454] T b1299a3340954c93b69e56dbccaca18e P bb456655a12a41039f76160cfda886ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e5fefbbe4064049919e70cf86eb9757 in term 1.
I20260807 22:56:22.978480 28985 raft_consensus.cc:2454] T b1299a3340954c93b69e56dbccaca18e P 856a406170f44f378bfd5638cf75ee7a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e5fefbbe4064049919e70cf86eb9757 in term 1.
I20260807 22:56:22.978874 29094 leader_election.cc:302] T b1299a3340954c93b69e56dbccaca18e P 9e5fefbbe4064049919e70cf86eb9757 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 856a406170f44f378bfd5638cf75ee7a, 9e5fefbbe4064049919e70cf86eb9757; no voters:
I20260807 22:56:22.979190 29600 raft_consensus.cc:2781] T b1299a3340954c93b69e56dbccaca18e P 9e5fefbbe4064049919e70cf86eb9757 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:56:22.979358 29600 raft_consensus.cc:686] T b1299a3340954c93b69e56dbccaca18e P 9e5fefbbe4064049919e70cf86eb9757 [term 1 LEADER]: Becoming Leader. State: Replica: 9e5fefbbe4064049919e70cf86eb9757, State: Running, Role: LEADER
I20260807 22:56:22.979628 29600 consensus_queue.cc:237] T b1299a3340954c93b69e56dbccaca18e P 9e5fefbbe4064049919e70cf86eb9757 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "856a406170f44f378bfd5638cf75ee7a" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44541 } } peers { permanent_uuid: "9e5fefbbe4064049919e70cf86eb9757" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 45043 } } peers { permanent_uuid: "bb456655a12a41039f76160cfda886ea" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 45619 } }
I20260807 22:56:22.983047 28725 catalog_manager.cc:5495] T b1299a3340954c93b69e56dbccaca18e P 9e5fefbbe4064049919e70cf86eb9757 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9e5fefbbe4064049919e70cf86eb9757 (127.25.94.2). New cstate: current_term: 1 leader_uuid: "9e5fefbbe4064049919e70cf86eb9757" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "856a406170f44f378bfd5638cf75ee7a" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44541 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e5fefbbe4064049919e70cf86eb9757" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 45043 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb456655a12a41039f76160cfda886ea" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 45619 } health_report { overall_health: UNKNOWN } } }
I20260807 22:56:23.007494 29669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:23.001161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54272 (local address 127.25.94.2:45043)
0807 22:56:23.002146 (+ 985us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:23.002150 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:23.002186 (+ 36us) server_negotiation.cc:407] Connection header received
0807 22:56:23.002241 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:23.002244 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:23.002308 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:23.002394 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 22:56:23.002750 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:23.003999 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:23.005587 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:23.006205 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:23.006305 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:23.006818 (+ 513us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 22:56:23.006951 (+ 133us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 22:56:23.007022 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:23.007152 (+ 130us) server_negotiation.cc:299] Negotiation successful
0807 22:56:23.007214 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":771,"thread_start_us":564,"threads_started":1}
I20260807 22:56:23.022099 29669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:23.017089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.2:53495 (local address 127.25.94.2:45043)
0807 22:56:23.017343 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:23.017352 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:23.017589 (+ 237us) server_negotiation.cc:407] Connection header received
0807 22:56:23.017846 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:23.017849 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:23.017904 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:23.017990 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 22:56:23.018368 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:23.019185 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:23.020488 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:23.020885 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:23.021024 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:23.021440 (+ 416us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 22:56:23.021530 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 22:56:23.021625 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:23.021730 (+ 105us) server_negotiation.cc:299] Negotiation successful
0807 22:56:23.021801 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260807 22:56:23.022198 29675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:23.016782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.2:45043 (local address 127.25.94.2:53495)
0807 22:56:23.017494 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0807 22:56:23.017509 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 22:56:23.017663 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:56:23.018001 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:56:23.018005 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:56:23.018015 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0807 22:56:23.018230 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:23.018246 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:23.019323 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:23.019328 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:23.020350 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:23.020361 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:23.021040 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:23.021045 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:23.021245 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:23.021291 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0807 22:56:23.021628 (+ 337us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0807 22:56:23.021641 (+ 13us) client_negotiation.cc:769] Sending connection context
0807 22:56:23.021715 (+ 74us) client_negotiation.cc:240] Negotiation successful
0807 22:56:23.021837 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":381,"threads_started":1}
I20260807 22:56:23.030529 29406 raft_consensus.cc:1261] T b1299a3340954c93b69e56dbccaca18e P bb456655a12a41039f76160cfda886ea [term 1 FOLLOWER]: Refusing update from remote peer 9e5fefbbe4064049919e70cf86eb9757: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 22:56:23.030608 28985 raft_consensus.cc:1261] T b1299a3340954c93b69e56dbccaca18e P 856a406170f44f378bfd5638cf75ee7a [term 1 FOLLOWER]: Refusing update from remote peer 9e5fefbbe4064049919e70cf86eb9757: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 22:56:23.031055 29600 consensus_queue.cc:1035] T b1299a3340954c93b69e56dbccaca18e P 9e5fefbbe4064049919e70cf86eb9757 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb456655a12a41039f76160cfda886ea" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 45619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 22:56:23.031352 29630 consensus_queue.cc:1035] T b1299a3340954c93b69e56dbccaca18e P 9e5fefbbe4064049919e70cf86eb9757 [LEADER]: Connected to new peer: Peer: permanent_uuid: "856a406170f44f378bfd5638cf75ee7a" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 22:56:23.089787 28725 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47734:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 22:56:23.090026 28725 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47734:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 22:56:23.093905 28725 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1c2a3363ee8640eeb7d932cc0d4c104b: Sending DeleteTablet for 3 replicas of tablet b1299a3340954c93b69e56dbccaca18e
I20260807 22:56:23.095115 28950 tablet_service.cc:1514] Processing DeleteTablet for tablet b1299a3340954c93b69e56dbccaca18e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:56:23 UTC) from {username='kudu', principal='kudu/127.25.94.62@KRBTEST.COM'} at 127.0.0.1:51654
I20260807 22:56:23.095302 29161 tablet_service.cc:1514] Processing DeleteTablet for tablet b1299a3340954c93b69e56dbccaca18e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:56:23 UTC) from {username='kudu', principal='kudu/127.25.94.62@KRBTEST.COM'} at 127.0.0.1:55172
I20260807 22:56:23.095646 29636 tablet_replica.cc:330] T b1299a3340954c93b69e56dbccaca18e P 9e5fefbbe4064049919e70cf86eb9757: stopping tablet replica
I20260807 22:56:23.095932 29636 raft_consensus.cc:2229] T b1299a3340954c93b69e56dbccaca18e P 9e5fefbbe4064049919e70cf86eb9757 [term 1 LEADER]: Raft consensus shutting down.
I20260807 22:56:23.095963 29362 tablet_service.cc:1514] Processing DeleteTablet for tablet b1299a3340954c93b69e56dbccaca18e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:56:23 UTC) from {username='kudu', principal='kudu/127.25.94.62@KRBTEST.COM'} at 127.0.0.1:59690
I20260807 22:56:23.096367 29638 tablet_replica.cc:330] T b1299a3340954c93b69e56dbccaca18e P bb456655a12a41039f76160cfda886ea: stopping tablet replica
I20260807 22:56:23.096573 29636 raft_consensus.cc:2258] T b1299a3340954c93b69e56dbccaca18e P 9e5fefbbe4064049919e70cf86eb9757 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 22:56:23.096698 29638 raft_consensus.cc:2229] T b1299a3340954c93b69e56dbccaca18e P bb456655a12a41039f76160cfda886ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 22:56:23.096987 29638 raft_consensus.cc:2258] T b1299a3340954c93b69e56dbccaca18e P bb456655a12a41039f76160cfda886ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 22:56:23.097318 28725 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47734:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260807 22:56:23.095536 29635 tablet_replica.cc:330] T b1299a3340954c93b69e56dbccaca18e P 856a406170f44f378bfd5638cf75ee7a: stopping tablet replica
I20260807 22:56:23.097879 29635 raft_consensus.cc:2229] T b1299a3340954c93b69e56dbccaca18e P 856a406170f44f378bfd5638cf75ee7a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 22:56:23.098094 29635 raft_consensus.cc:2258] T b1299a3340954c93b69e56dbccaca18e P 856a406170f44f378bfd5638cf75ee7a [term 1 FOLLOWER]: Raft consensus is shut down!
W20260807 22:56:23.098230 28725 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 22:56:23.098326 29638 ts_tablet_manager.cc:1905] T b1299a3340954c93b69e56dbccaca18e P bb456655a12a41039f76160cfda886ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 22:56:23.099261 29635 ts_tablet_manager.cc:1905] T b1299a3340954c93b69e56dbccaca18e P 856a406170f44f378bfd5638cf75ee7a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 22:56:23.099340 29636 ts_tablet_manager.cc:1905] T b1299a3340954c93b69e56dbccaca18e P 9e5fefbbe4064049919e70cf86eb9757: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 22:56:23.106660 28950 tablet_service.cc:1467] Processing CreateTablet for tablet 16099542d7834a2b862bfe997634a2d5 (DEFAULT_TABLE table=test-table [id=b74d93c7405c4bbb986ebcfdee385b40]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:56:23.107182 28950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16099542d7834a2b862bfe997634a2d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:56:23.107604 29635 ts_tablet_manager.cc:1918] T b1299a3340954c93b69e56dbccaca18e P 856a406170f44f378bfd5638cf75ee7a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 22:56:23.107771 29161 tablet_service.cc:1467] Processing CreateTablet for tablet 16099542d7834a2b862bfe997634a2d5 (DEFAULT_TABLE table=test-table [id=b74d93c7405c4bbb986ebcfdee385b40]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:56:23.107901 29635 log.cc:1196] T b1299a3340954c93b69e56dbccaca18e P 856a406170f44f378bfd5638cf75ee7a: Deleting WAL directory at /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786143361779964-25976-0/minicluster-data/ts-0/wal/wals/b1299a3340954c93b69e56dbccaca18e
I20260807 22:56:23.108354 29161 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16099542d7834a2b862bfe997634a2d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:56:23.108623 29635 ts_tablet_manager.cc:1939] T b1299a3340954c93b69e56dbccaca18e P 856a406170f44f378bfd5638cf75ee7a: Deleting consensus metadata
I20260807 22:56:23.109642 29362 tablet_service.cc:1467] Processing CreateTablet for tablet 16099542d7834a2b862bfe997634a2d5 (DEFAULT_TABLE table=test-table [id=b74d93c7405c4bbb986ebcfdee385b40]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:56:23.110085 29362 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16099542d7834a2b862bfe997634a2d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:56:23.113227 29636 ts_tablet_manager.cc:1918] T b1299a3340954c93b69e56dbccaca18e P 9e5fefbbe4064049919e70cf86eb9757: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 22:56:23.113431 29636 log.cc:1196] T b1299a3340954c93b69e56dbccaca18e P 9e5fefbbe4064049919e70cf86eb9757: Deleting WAL directory at /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786143361779964-25976-0/minicluster-data/ts-1/wal/wals/b1299a3340954c93b69e56dbccaca18e
I20260807 22:56:23.115615 29636 ts_tablet_manager.cc:1939] T b1299a3340954c93b69e56dbccaca18e P 9e5fefbbe4064049919e70cf86eb9757: Deleting consensus metadata
I20260807 22:56:23.116815 29655 tablet_bootstrap.cc:492] T 16099542d7834a2b862bfe997634a2d5 P 9e5fefbbe4064049919e70cf86eb9757: Bootstrap starting.
I20260807 22:56:23.116946 28700 catalog_manager.cc:4841] TS 9e5fefbbe4064049919e70cf86eb9757 (127.25.94.2:45043): tablet b1299a3340954c93b69e56dbccaca18e (table test-table [id=62992ca864d3442b929fad67b8276dff]) successfully deleted
I20260807 22:56:23.119659 29655 tablet_bootstrap.cc:654] T 16099542d7834a2b862bfe997634a2d5 P 9e5fefbbe4064049919e70cf86eb9757: Neither blocks nor log segments found. Creating new log.
I20260807 22:56:23.120217 28698 catalog_manager.cc:4841] TS 856a406170f44f378bfd5638cf75ee7a (127.25.94.1:44541): tablet b1299a3340954c93b69e56dbccaca18e (table test-table [id=62992ca864d3442b929fad67b8276dff]) successfully deleted
I20260807 22:56:23.125888 29656 tablet_bootstrap.cc:492] T 16099542d7834a2b862bfe997634a2d5 P bb456655a12a41039f76160cfda886ea: Bootstrap starting.
I20260807 22:56:23.128293 29656 tablet_bootstrap.cc:654] T 16099542d7834a2b862bfe997634a2d5 P bb456655a12a41039f76160cfda886ea: Neither blocks nor log segments found. Creating new log.
I20260807 22:56:23.134773 29638 ts_tablet_manager.cc:1918] T b1299a3340954c93b69e56dbccaca18e P bb456655a12a41039f76160cfda886ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 22:56:23.139338 29658 tablet_bootstrap.cc:492] T 16099542d7834a2b862bfe997634a2d5 P 856a406170f44f378bfd5638cf75ee7a: Bootstrap starting.
I20260807 22:56:23.139609 29638 log.cc:1196] T b1299a3340954c93b69e56dbccaca18e P bb456655a12a41039f76160cfda886ea: Deleting WAL directory at /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786143361779964-25976-0/minicluster-data/ts-2/wal/wals/b1299a3340954c93b69e56dbccaca18e
I20260807 22:56:23.138886 29656 tablet_bootstrap.cc:492] T 16099542d7834a2b862bfe997634a2d5 P bb456655a12a41039f76160cfda886ea: No bootstrap required, opened a new log
I20260807 22:56:23.139854 29656 ts_tablet_manager.cc:1397] T 16099542d7834a2b862bfe997634a2d5 P bb456655a12a41039f76160cfda886ea: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.005s
I20260807 22:56:23.140194 29638 ts_tablet_manager.cc:1939] T b1299a3340954c93b69e56dbccaca18e P bb456655a12a41039f76160cfda886ea: Deleting consensus metadata
I20260807 22:56:23.140764 29656 raft_consensus.cc:348] T 16099542d7834a2b862bfe997634a2d5 P bb456655a12a41039f76160cfda886ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "856a406170f44f378bfd5638cf75ee7a" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44541 } } peers { permanent_uuid: "9e5fefbbe4064049919e70cf86eb9757" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 45043 } } peers { permanent_uuid: "bb456655a12a41039f76160cfda886ea" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 45619 } }
I20260807 22:56:23.141041 29656 raft_consensus.cc:374] T 16099542d7834a2b862bfe997634a2d5 P bb456655a12a41039f76160cfda886ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:56:23.141480 28700 catalog_manager.cc:4841] TS bb456655a12a41039f76160cfda886ea (127.25.94.3:45619): tablet b1299a3340954c93b69e56dbccaca18e (table test-table [id=62992ca864d3442b929fad67b8276dff]) successfully deleted
I20260807 22:56:23.141559 29658 tablet_bootstrap.cc:654] T 16099542d7834a2b862bfe997634a2d5 P 856a406170f44f378bfd5638cf75ee7a: Neither blocks nor log segments found. Creating new log.
I20260807 22:56:23.141800 29656 raft_consensus.cc:729] T 16099542d7834a2b862bfe997634a2d5 P bb456655a12a41039f76160cfda886ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb456655a12a41039f76160cfda886ea, State: Initialized, Role: FOLLOWER
I20260807 22:56:23.142329 29656 consensus_queue.cc:260] T 16099542d7834a2b862bfe997634a2d5 P bb456655a12a41039f76160cfda886ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "856a406170f44f378bfd5638cf75ee7a" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44541 } } peers { permanent_uuid: "9e5fefbbe4064049919e70cf86eb9757" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 45043 } } peers { permanent_uuid: "bb456655a12a41039f76160cfda886ea" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 45619 } }
I20260807 22:56:23.143496 29656 ts_tablet_manager.cc:1428] T 16099542d7834a2b862bfe997634a2d5 P bb456655a12a41039f76160cfda886ea: Time spent starting tablet: real 0.004s user 0.001s sys 0.002s
I20260807 22:56:23.144443 29655 tablet_bootstrap.cc:492] T 16099542d7834a2b862bfe997634a2d5 P 9e5fefbbe4064049919e70cf86eb9757: No bootstrap required, opened a new log
I20260807 22:56:23.145033 29655 ts_tablet_manager.cc:1397] T 16099542d7834a2b862bfe997634a2d5 P 9e5fefbbe4064049919e70cf86eb9757: Time spent bootstrapping tablet: real 0.028s user 0.016s sys 0.008s
I20260807 22:56:23.146402 29655 raft_consensus.cc:348] T 16099542d7834a2b862bfe997634a2d5 P 9e5fefbbe4064049919e70cf86eb9757 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "856a406170f44f378bfd5638cf75ee7a" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44541 } } peers { permanent_uuid: "9e5fefbbe4064049919e70cf86eb9757" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 45043 } } peers { permanent_uuid: "bb456655a12a41039f76160cfda886ea" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 45619 } }
I20260807 22:56:23.146804 29655 raft_consensus.cc:374] T 16099542d7834a2b862bfe997634a2d5 P 9e5fefbbe4064049919e70cf86eb9757 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:56:23.146914 29655 raft_consensus.cc:729] T 16099542d7834a2b862bfe997634a2d5 P 9e5fefbbe4064049919e70cf86eb9757 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e5fefbbe4064049919e70cf86eb9757, State: Initialized, Role: FOLLOWER
I20260807 22:56:23.147192 29655 consensus_queue.cc:260] T 16099542d7834a2b862bfe997634a2d5 P 9e5fefbbe4064049919e70cf86eb9757 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "856a406170f44f378bfd5638cf75ee7a" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44541 } } peers { permanent_uuid: "9e5fefbbe4064049919e70cf86eb9757" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 45043 } } peers { permanent_uuid: "bb456655a12a41039f76160cfda886ea" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 45619 } }
I20260807 22:56:23.147921 29655 ts_tablet_manager.cc:1428] T 16099542d7834a2b862bfe997634a2d5 P 9e5fefbbe4064049919e70cf86eb9757: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260807 22:56:23.148757 29658 tablet_bootstrap.cc:492] T 16099542d7834a2b862bfe997634a2d5 P 856a406170f44f378bfd5638cf75ee7a: No bootstrap required, opened a new log
I20260807 22:56:23.148976 29658 ts_tablet_manager.cc:1397] T 16099542d7834a2b862bfe997634a2d5 P 856a406170f44f378bfd5638cf75ee7a: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.005s
I20260807 22:56:23.149844 29658 raft_consensus.cc:348] T 16099542d7834a2b862bfe997634a2d5 P 856a406170f44f378bfd5638cf75ee7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "856a406170f44f378bfd5638cf75ee7a" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44541 } } peers { permanent_uuid: "9e5fefbbe4064049919e70cf86eb9757" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 45043 } } peers { permanent_uuid: "bb456655a12a41039f76160cfda886ea" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 45619 } }
I20260807 22:56:23.150245 29658 raft_consensus.cc:374] T 16099542d7834a2b862bfe997634a2d5 P 856a406170f44f378bfd5638cf75ee7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:56:23.150381 29658 raft_consensus.cc:729] T 16099542d7834a2b862bfe997634a2d5 P 856a406170f44f378bfd5638cf75ee7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 856a406170f44f378bfd5638cf75ee7a, State: Initialized, Role: FOLLOWER
I20260807 22:56:23.150653 29658 consensus_queue.cc:260] T 16099542d7834a2b862bfe997634a2d5 P 856a406170f44f378bfd5638cf75ee7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "856a406170f44f378bfd5638cf75ee7a" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44541 } } peers { permanent_uuid: "9e5fefbbe4064049919e70cf86eb9757" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 45043 } } peers { permanent_uuid: "bb456655a12a41039f76160cfda886ea" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 45619 } }
I20260807 22:56:23.151304 29658 ts_tablet_manager.cc:1428] T 16099542d7834a2b862bfe997634a2d5 P 856a406170f44f378bfd5638cf75ee7a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260807 22:56:23.478852 29624 raft_consensus.cc:482] T 16099542d7834a2b862bfe997634a2d5 P 856a406170f44f378bfd5638cf75ee7a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 22:56:23.479152 29624 raft_consensus.cc:504] T 16099542d7834a2b862bfe997634a2d5 P 856a406170f44f378bfd5638cf75ee7a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "856a406170f44f378bfd5638cf75ee7a" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44541 } } peers { permanent_uuid: "9e5fefbbe4064049919e70cf86eb9757" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 45043 } } peers { permanent_uuid: "bb456655a12a41039f76160cfda886ea" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 45619 } }
I20260807 22:56:23.480072 29624 leader_election.cc:288] T 16099542d7834a2b862bfe997634a2d5 P 856a406170f44f378bfd5638cf75ee7a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9e5fefbbe4064049919e70cf86eb9757 (127.25.94.2:45043), bb456655a12a41039f76160cfda886ea (127.25.94.3:45619)
I20260807 22:56:23.480921 29407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16099542d7834a2b862bfe997634a2d5" candidate_uuid: "856a406170f44f378bfd5638cf75ee7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb456655a12a41039f76160cfda886ea" is_pre_election: true
I20260807 22:56:23.480922 29192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16099542d7834a2b862bfe997634a2d5" candidate_uuid: "856a406170f44f378bfd5638cf75ee7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e5fefbbe4064049919e70cf86eb9757" is_pre_election: true
I20260807 22:56:23.481356 29407 raft_consensus.cc:2454] T 16099542d7834a2b862bfe997634a2d5 P bb456655a12a41039f76160cfda886ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 856a406170f44f378bfd5638cf75ee7a in term 0.
I20260807 22:56:23.481341 29192 raft_consensus.cc:2454] T 16099542d7834a2b862bfe997634a2d5 P 9e5fefbbe4064049919e70cf86eb9757 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 856a406170f44f378bfd5638cf75ee7a in term 0.
I20260807 22:56:23.481854 28871 leader_election.cc:302] T 16099542d7834a2b862bfe997634a2d5 P 856a406170f44f378bfd5638cf75ee7a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 856a406170f44f378bfd5638cf75ee7a, bb456655a12a41039f76160cfda886ea; no voters:
I20260807 22:56:23.482224 29624 raft_consensus.cc:2781] T 16099542d7834a2b862bfe997634a2d5 P 856a406170f44f378bfd5638cf75ee7a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 22:56:23.482390 29624 raft_consensus.cc:482] T 16099542d7834a2b862bfe997634a2d5 P 856a406170f44f378bfd5638cf75ee7a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 22:56:23.482496 29624 raft_consensus.cc:3037] T 16099542d7834a2b862bfe997634a2d5 P 856a406170f44f378bfd5638cf75ee7a [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:56:23.486027 29624 raft_consensus.cc:504] T 16099542d7834a2b862bfe997634a2d5 P 856a406170f44f378bfd5638cf75ee7a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "856a406170f44f378bfd5638cf75ee7a" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44541 } } peers { permanent_uuid: "9e5fefbbe4064049919e70cf86eb9757" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 45043 } } peers { permanent_uuid: "bb456655a12a41039f76160cfda886ea" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 45619 } }
I20260807 22:56:23.486820 29624 leader_election.cc:288] T 16099542d7834a2b862bfe997634a2d5 P 856a406170f44f378bfd5638cf75ee7a [CANDIDATE]: Term 1 election: Requested vote from peers 9e5fefbbe4064049919e70cf86eb9757 (127.25.94.2:45043), bb456655a12a41039f76160cfda886ea (127.25.94.3:45619)
I20260807 22:56:23.487650 29192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16099542d7834a2b862bfe997634a2d5" candidate_uuid: "856a406170f44f378bfd5638cf75ee7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e5fefbbe4064049919e70cf86eb9757"
I20260807 22:56:23.487679 29407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16099542d7834a2b862bfe997634a2d5" candidate_uuid: "856a406170f44f378bfd5638cf75ee7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb456655a12a41039f76160cfda886ea"
I20260807 22:56:23.487932 29192 raft_consensus.cc:3037] T 16099542d7834a2b862bfe997634a2d5 P 9e5fefbbe4064049919e70cf86eb9757 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:56:23.488065 29407 raft_consensus.cc:3037] T 16099542d7834a2b862bfe997634a2d5 P bb456655a12a41039f76160cfda886ea [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:56:23.491793 29407 raft_consensus.cc:2454] T 16099542d7834a2b862bfe997634a2d5 P bb456655a12a41039f76160cfda886ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 856a406170f44f378bfd5638cf75ee7a in term 1.
I20260807 22:56:23.491848 29192 raft_consensus.cc:2454] T 16099542d7834a2b862bfe997634a2d5 P 9e5fefbbe4064049919e70cf86eb9757 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 856a406170f44f378bfd5638cf75ee7a in term 1.
I20260807 22:56:23.492345 28871 leader_election.cc:302] T 16099542d7834a2b862bfe997634a2d5 P 856a406170f44f378bfd5638cf75ee7a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 856a406170f44f378bfd5638cf75ee7a, 9e5fefbbe4064049919e70cf86eb9757; no voters:
I20260807 22:56:23.492813 29624 raft_consensus.cc:2781] T 16099542d7834a2b862bfe997634a2d5 P 856a406170f44f378bfd5638cf75ee7a [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:56:23.493341 29624 raft_consensus.cc:686] T 16099542d7834a2b862bfe997634a2d5 P 856a406170f44f378bfd5638cf75ee7a [term 1 LEADER]: Becoming Leader. State: Replica: 856a406170f44f378bfd5638cf75ee7a, State: Running, Role: LEADER
I20260807 22:56:23.493754 29624 consensus_queue.cc:237] T 16099542d7834a2b862bfe997634a2d5 P 856a406170f44f378bfd5638cf75ee7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "856a406170f44f378bfd5638cf75ee7a" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44541 } } peers { permanent_uuid: "9e5fefbbe4064049919e70cf86eb9757" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 45043 } } peers { permanent_uuid: "bb456655a12a41039f76160cfda886ea" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 45619 } }
I20260807 22:56:23.497130 28725 catalog_manager.cc:5495] T 16099542d7834a2b862bfe997634a2d5 P 856a406170f44f378bfd5638cf75ee7a reported cstate change: term changed from 0 to 1, leader changed from <none> to 856a406170f44f378bfd5638cf75ee7a (127.25.94.1). New cstate: current_term: 1 leader_uuid: "856a406170f44f378bfd5638cf75ee7a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "856a406170f44f378bfd5638cf75ee7a" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44541 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e5fefbbe4064049919e70cf86eb9757" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 45043 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb456655a12a41039f76160cfda886ea" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 45619 } health_report { overall_health: UNKNOWN } } }
I20260807 22:56:23.615118 29700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:23.590600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59042 (local address 127.25.94.1:44541)
0807 22:56:23.607969 (+ 17369us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:23.607975 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:23.608001 (+ 26us) server_negotiation.cc:407] Connection header received
0807 22:56:23.608070 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:23.608076 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:23.608200 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:23.608303 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 22:56:23.608908 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:23.610427 (+ 1519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:23.612019 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:23.612786 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:23.612981 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:23.613609 (+ 628us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 22:56:23.613863 (+ 254us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 22:56:23.614029 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:23.614173 (+ 144us) server_negotiation.cc:299] Negotiation successful
0807 22:56:23.614328 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17232,"thread_start_us":17907,"threads_started":1}
I20260807 22:56:23.625793 29406 raft_consensus.cc:1261] T 16099542d7834a2b862bfe997634a2d5 P bb456655a12a41039f76160cfda886ea [term 1 FOLLOWER]: Refusing update from remote peer 856a406170f44f378bfd5638cf75ee7a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 22:56:23.625911 29192 raft_consensus.cc:1261] T 16099542d7834a2b862bfe997634a2d5 P 9e5fefbbe4064049919e70cf86eb9757 [term 1 FOLLOWER]: Refusing update from remote peer 856a406170f44f378bfd5638cf75ee7a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 22:56:23.626422 29687 consensus_queue.cc:1035] T 16099542d7834a2b862bfe997634a2d5 P 856a406170f44f378bfd5638cf75ee7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb456655a12a41039f76160cfda886ea" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 45619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 22:56:23.626767 29624 consensus_queue.cc:1035] T 16099542d7834a2b862bfe997634a2d5 P 856a406170f44f378bfd5638cf75ee7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e5fefbbe4064049919e70cf86eb9757" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 45043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260807 22:56:23.650377 29693 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260807 22:56:25.245467 28727 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47734:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 22:56:25.245801 28727 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47734:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 22:56:25.258126 28727 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1c2a3363ee8640eeb7d932cc0d4c104b: Sending DeleteTablet for 3 replicas of tablet 16099542d7834a2b862bfe997634a2d5
I20260807 22:56:25.259438 28950 tablet_service.cc:1514] Processing DeleteTablet for tablet 16099542d7834a2b862bfe997634a2d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:56:25 UTC) from {username='kudu', principal='kudu/127.25.94.62@KRBTEST.COM'} at 127.0.0.1:51654
I20260807 22:56:25.259583 29362 tablet_service.cc:1514] Processing DeleteTablet for tablet 16099542d7834a2b862bfe997634a2d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:56:25 UTC) from {username='kudu', principal='kudu/127.25.94.62@KRBTEST.COM'} at 127.0.0.1:59690
I20260807 22:56:25.260052 29161 tablet_service.cc:1514] Processing DeleteTablet for tablet 16099542d7834a2b862bfe997634a2d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:56:25 UTC) from {username='kudu', principal='kudu/127.25.94.62@KRBTEST.COM'} at 127.0.0.1:55172
I20260807 22:56:25.261344 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 28843
I20260807 22:56:25.271263 29752 tablet_replica.cc:330] T 16099542d7834a2b862bfe997634a2d5 P 9e5fefbbe4064049919e70cf86eb9757: stopping tablet replica
I20260807 22:56:25.275054 29752 raft_consensus.cc:2229] T 16099542d7834a2b862bfe997634a2d5 P 9e5fefbbe4064049919e70cf86eb9757 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 22:56:25.276312 29752 raft_consensus.cc:2258] T 16099542d7834a2b862bfe997634a2d5 P 9e5fefbbe4064049919e70cf86eb9757 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 22:56:25.277496 29750 tablet_replica.cc:330] T 16099542d7834a2b862bfe997634a2d5 P bb456655a12a41039f76160cfda886ea: stopping tablet replica
I20260807 22:56:25.278218 29750 raft_consensus.cc:2229] T 16099542d7834a2b862bfe997634a2d5 P bb456655a12a41039f76160cfda886ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 22:56:25.278390 29752 ts_tablet_manager.cc:1905] T 16099542d7834a2b862bfe997634a2d5 P 9e5fefbbe4064049919e70cf86eb9757: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 22:56:25.278861 29750 raft_consensus.cc:2258] T 16099542d7834a2b862bfe997634a2d5 P bb456655a12a41039f76160cfda886ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 22:56:25.279968 29748 tablet_replica.cc:330] T 16099542d7834a2b862bfe997634a2d5 P 856a406170f44f378bfd5638cf75ee7a: stopping tablet replica
I20260807 22:56:25.280402 29748 raft_consensus.cc:2229] T 16099542d7834a2b862bfe997634a2d5 P 856a406170f44f378bfd5638cf75ee7a [term 1 LEADER]: Raft consensus shutting down.
I20260807 22:56:25.280962 29750 ts_tablet_manager.cc:1905] T 16099542d7834a2b862bfe997634a2d5 P bb456655a12a41039f76160cfda886ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 22:56:25.281086 29748 raft_consensus.cc:2258] T 16099542d7834a2b862bfe997634a2d5 P 856a406170f44f378bfd5638cf75ee7a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 22:56:25.282380 29748 ts_tablet_manager.cc:1905] T 16099542d7834a2b862bfe997634a2d5 P 856a406170f44f378bfd5638cf75ee7a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 22:56:25.288537 29750 ts_tablet_manager.cc:1918] T 16099542d7834a2b862bfe997634a2d5 P bb456655a12a41039f76160cfda886ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260807 22:56:25.288795 29750 log.cc:1196] T 16099542d7834a2b862bfe997634a2d5 P bb456655a12a41039f76160cfda886ea: Deleting WAL directory at /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786143361779964-25976-0/minicluster-data/ts-2/wal/wals/16099542d7834a2b862bfe997634a2d5
I20260807 22:56:25.289474 29752 ts_tablet_manager.cc:1918] T 16099542d7834a2b862bfe997634a2d5 P 9e5fefbbe4064049919e70cf86eb9757: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260807 22:56:25.289816 29750 ts_tablet_manager.cc:1939] T 16099542d7834a2b862bfe997634a2d5 P bb456655a12a41039f76160cfda886ea: Deleting consensus metadata
I20260807 22:56:25.289897 29752 log.cc:1196] T 16099542d7834a2b862bfe997634a2d5 P 9e5fefbbe4064049919e70cf86eb9757: Deleting WAL directory at /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786143361779964-25976-0/minicluster-data/ts-1/wal/wals/16099542d7834a2b862bfe997634a2d5
I20260807 22:56:25.290800 29752 ts_tablet_manager.cc:1939] T 16099542d7834a2b862bfe997634a2d5 P 9e5fefbbe4064049919e70cf86eb9757: Deleting consensus metadata
I20260807 22:56:25.291352 28700 catalog_manager.cc:4841] TS bb456655a12a41039f76160cfda886ea (127.25.94.3:45619): tablet 16099542d7834a2b862bfe997634a2d5 (table test-table [id=b74d93c7405c4bbb986ebcfdee385b40]) successfully deleted
I20260807 22:56:25.292680 28700 catalog_manager.cc:4841] TS 9e5fefbbe4064049919e70cf86eb9757 (127.25.94.2:45043): tablet 16099542d7834a2b862bfe997634a2d5 (table test-table [id=b74d93c7405c4bbb986ebcfdee385b40]) successfully deleted
I20260807 22:56:25.293540 29748 ts_tablet_manager.cc:1918] T 16099542d7834a2b862bfe997634a2d5 P 856a406170f44f378bfd5638cf75ee7a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260807 22:56:25.293656 29748 log.cc:1196] T 16099542d7834a2b862bfe997634a2d5 P 856a406170f44f378bfd5638cf75ee7a: Deleting WAL directory at /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786143361779964-25976-0/minicluster-data/ts-0/wal/wals/16099542d7834a2b862bfe997634a2d5
I20260807 22:56:25.294165 29748 ts_tablet_manager.cc:1939] T 16099542d7834a2b862bfe997634a2d5 P 856a406170f44f378bfd5638cf75ee7a: Deleting consensus metadata
I20260807 22:56:25.295159 28698 catalog_manager.cc:4841] TS 856a406170f44f378bfd5638cf75ee7a (127.25.94.1:44541): tablet 16099542d7834a2b862bfe997634a2d5 (table test-table [id=b74d93c7405c4bbb986ebcfdee385b40]) successfully deleted
Aug 07 22:56:25 dist-test-slave-z8x9 krb5kdc[28646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143382, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.25.94.1@KRBTEST.COM
I20260807 22:56:25.313432 29756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:25.262248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59048 (local address 127.25.94.1:44541)
0807 22:56:25.283296 (+ 21048us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:25.283304 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:25.283359 (+ 55us) server_negotiation.cc:407] Connection header received
0807 22:56:25.283423 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:25.283428 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:25.283512 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:25.283662 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:56:25.290956 (+ 7294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:25.292738 (+ 1782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:25.294430 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:25.295210 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:25.295378 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:25.304086 (+ 8708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:56:25.304105 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:56:25.304122 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:56:25.304240 (+ 118us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:56:25.309118 (+ 4878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:25.310313 (+ 1195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:25.310322 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:25.310324 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:25.310667 (+ 343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:25.311260 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:25.311264 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:25.311266 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:25.311808 (+ 542us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:56:25.312123 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:25.312531 (+ 408us) server_negotiation.cc:299] Negotiation successful
0807 22:56:25.312746 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20909,"thread_start_us":20796,"threads_started":1}
W20260807 22:56:25.314347 29053 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59048
W20260807 22:56:25.315294 25976 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 28843: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 22:56:25.315661 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 28843
I20260807 22:56:25.347100 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 29064
Aug 07 22:56:25 dist-test-slave-z8x9 krb5kdc[28646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143382, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.25.94.2@KRBTEST.COM
I20260807 22:56:25.367888 29760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:25.348171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54282 (local address 127.25.94.2:45043)
0807 22:56:25.348827 (+ 656us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:25.348831 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:25.348848 (+ 17us) server_negotiation.cc:407] Connection header received
0807 22:56:25.348901 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:25.348905 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:25.348965 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:25.349081 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:56:25.350897 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:25.351696 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:25.354008 (+ 2312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:25.354527 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:25.354640 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:25.361379 (+ 6739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:56:25.361409 (+ 30us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:56:25.361427 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:56:25.361467 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:56:25.364792 (+ 3325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:25.365683 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:25.365691 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:25.365693 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:25.365842 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:25.366257 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:25.366262 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:25.366264 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:25.366639 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:56:25.366968 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:25.367273 (+ 305us) server_negotiation.cc:299] Negotiation successful
0807 22:56:25.367531 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":539,"thread_start_us":397,"threads_started":1}
W20260807 22:56:25.368566 29257 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54282
W20260807 22:56:25.369165 25976 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 29064: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 22:56:25.369355 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 29064
I20260807 22:56:25.402384 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 29269
Aug 07 22:56:25 dist-test-slave-z8x9 krb5kdc[28646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143382, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.25.94.3@KRBTEST.COM
I20260807 22:56:25.419826 29761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:25.403240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57046 (local address 127.25.94.3:45619)
0807 22:56:25.403807 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:25.403811 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:25.403827 (+ 16us) server_negotiation.cc:407] Connection header received
0807 22:56:25.403883 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:25.403886 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:25.403952 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:25.404077 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:56:25.405521 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:25.406303 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:25.408010 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:25.408464 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:25.408611 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:25.413598 (+ 4987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:56:25.413615 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:56:25.413632 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:56:25.413670 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:56:25.416761 (+ 3091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:25.417588 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:25.417605 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:25.417608 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:25.417756 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:25.418163 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:25.418169 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:25.418171 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:25.418554 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:56:25.418887 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:25.419210 (+ 323us) server_negotiation.cc:299] Negotiation successful
0807 22:56:25.419463 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":305,"threads_started":1}
W20260807 22:56:25.420590 29467 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57046
W20260807 22:56:25.421262 25976 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 29269: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 22:56:25.421419 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 29269
I20260807 22:56:25.454442 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 28675
I20260807 22:56:25.455729 28792 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 22:56:25.580384 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 28675
2026-08-07T22:56:25Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (9038 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786143361779964-25976-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 22:56:25 dist-test-slave-z8x9 krb5kdc[29767](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 22:56:25 dist-test-slave-z8x9 krb5kdc[29767](info): set up 2 sockets
Aug 07 22:56:25 dist-test-slave-z8x9 krb5kdc[29767](info): commencing operation
krb5kdc: starting...
W20260807 22:56:26.938236 25976 mini_kdc.cc:121] Time spent starting KDC: real 1.307s user 0.001s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786143361779964-25976-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786143361779964-25976-0/krb5kdc/test-user.keytab.
Aug 07 22:56:27 dist-test-slave-z8x9 krb5kdc[29767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143387, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T22:56:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T22:56:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.94.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.62.keytab.
Entry for principal kudu/127.25.94.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.62.keytab.
I20260807 22:56:27.171416 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786143361779964-25976-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786143361779964-25976-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786143361779964-25976-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.62:34547
--webserver_interface=127.25.94.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.62.keytab
--principal=kudu/127.25.94.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.20:44543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.94.62:34547
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786143361779964-25976-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786143361779964-25976-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786143361779964-25976-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:56:27.515877 29793 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:56:27.516263 29793 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:56:27.516433 29793 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:56:27.525475 29793 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 22:56:27.525627 29793 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:56:27.525734 29793 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:56:27.525846 29793 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 22:56:27.525930 29793 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 22:56:27.536164 29793 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:44543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786143361779964-25976-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786143361779964-25976-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.94.62:34547
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.62.keytab
--principal=kudu/127.25.94.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.62:34547
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.94.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786143361779964-25976-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 22:56:27.537992 29793 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:56:27.539517 29793 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:56:27.553807 29798 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:56:27.553766 29803 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:56:27.553742 29800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:56:27.554888 29793 server_base.cc:1034] running on GCE node
I20260807 22:56:27.555702 29793 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:56:27.557281 29793 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:56:27.558518 29793 hybrid_clock.cc:648] HybridClock initialized: now 1786143387558459 us; error 43 us; skew 500 ppm
I20260807 22:56:27.558843 29793 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 22:56:27 dist-test-slave-z8x9 krb5kdc[29767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143387, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:56:27.562165 29793 init.cc:377] Logged in from keytab as kudu/127.25.94.62@KRBTEST.COM (short username kudu)
I20260807 22:56:27.563862 29793 webserver.cc:460] Webserver started at http://127.25.94.62:42751/ using document root <none> and password file <none>
I20260807 22:56:27.564535 29793 fs_manager.cc:359] Metadata directory not provided
I20260807 22:56:27.564640 29793 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:56:27.564925 29793 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:56:27.568066 29793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786143361779964-25976-0/minicluster-data/master-0/data/instance:
uuid: "f7641d0236a24e8da5d450de8cf29d69"
format_stamp: "Formatted at 2026-08-07 22:56:27 on dist-test-slave-z8x9"
server_key: "908e57735be8296b2d7a52c443d7e8d1"
server_key_iv: "3c57d4437e9fd6ccfd490aa33d58775e"
server_key_version: "encryptionkey@0"
I20260807 22:56:27.568794 29793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786143361779964-25976-0/minicluster-data/master-0/wal/instance:
uuid: "f7641d0236a24e8da5d450de8cf29d69"
format_stamp: "Formatted at 2026-08-07 22:56:27 on dist-test-slave-z8x9"
server_key: "908e57735be8296b2d7a52c443d7e8d1"
server_key_iv: "3c57d4437e9fd6ccfd490aa33d58775e"
server_key_version: "encryptionkey@0"
I20260807 22:56:27.574240 29793 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260807 22:56:27.578030 29813 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:56:27.579466 29793 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260807 22:56:27.579684 29793 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786143361779964-25976-0/minicluster-data/master-0/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786143361779964-25976-0/minicluster-data/master-0/wal
uuid: "f7641d0236a24e8da5d450de8cf29d69"
format_stamp: "Formatted at 2026-08-07 22:56:27 on dist-test-slave-z8x9"
server_key: "908e57735be8296b2d7a52c443d7e8d1"
server_key_iv: "3c57d4437e9fd6ccfd490aa33d58775e"
server_key_version: "encryptionkey@0"
I20260807 22:56:27.579911 29793 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786143361779964-25976-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786143361779964-25976-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786143361779964-25976-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:56:27.598968 29793 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:56:27.603360 29793 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:56:27.603689 29793 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:56:27.623008 29793 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.62:34547
I20260807 22:56:27.623044 29906 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.62:34547 every 8 connection(s)
I20260807 22:56:27.624459 29793 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
I20260807 22:56:27.628270 29907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:56:27.633893 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 29793
I20260807 22:56:27.634151 25976 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786143361779964-25976-0/minicluster-data/master-0/wal/instance
I20260807 22:56:27.634804 25976 external_mini_cluster.cc:1428] Setting key baa47d5971c20341075078ee69fdc2fb
I20260807 22:56:27.640234 29907 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7641d0236a24e8da5d450de8cf29d69: Bootstrap starting.
I20260807 22:56:27.644049 29907 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f7641d0236a24e8da5d450de8cf29d69: Neither blocks nor log segments found. Creating new log.
I20260807 22:56:27.645318 29907 log.cc:824] T 00000000000000000000000000000000 P f7641d0236a24e8da5d450de8cf29d69: Log is configured to *not* fsync() on all Append() calls
I20260807 22:56:27.648581 29907 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7641d0236a24e8da5d450de8cf29d69: No bootstrap required, opened a new log
Aug 07 22:56:27 dist-test-slave-z8x9 krb5kdc[29767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143387, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.62@KRBTEST.COM
I20260807 22:56:27.659112 29907 raft_consensus.cc:348] T 00000000000000000000000000000000 P f7641d0236a24e8da5d450de8cf29d69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7641d0236a24e8da5d450de8cf29d69" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 34547 } }
I20260807 22:56:27.659575 29907 raft_consensus.cc:374] T 00000000000000000000000000000000 P f7641d0236a24e8da5d450de8cf29d69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:56:27.659682 29907 raft_consensus.cc:729] T 00000000000000000000000000000000 P f7641d0236a24e8da5d450de8cf29d69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7641d0236a24e8da5d450de8cf29d69, State: Initialized, Role: FOLLOWER
I20260807 22:56:27.660359 29907 consensus_queue.cc:260] T 00000000000000000000000000000000 P f7641d0236a24e8da5d450de8cf29d69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7641d0236a24e8da5d450de8cf29d69" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 34547 } }
I20260807 22:56:27.660657 29907 raft_consensus.cc:388] T 00000000000000000000000000000000 P f7641d0236a24e8da5d450de8cf29d69 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 22:56:27.660828 29907 raft_consensus.cc:482] T 00000000000000000000000000000000 P f7641d0236a24e8da5d450de8cf29d69 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 22:56:27.660880 29913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:27.636883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60788 (local address 127.25.94.62:34547)
0807 22:56:27.637833 (+ 950us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:27.637849 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:27.637901 (+ 52us) server_negotiation.cc:407] Connection header received
0807 22:56:27.640326 (+ 2425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:27.640357 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:27.640847 (+ 490us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:27.641312 (+ 465us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:56:27.642692 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:27.644708 (+ 2016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:27.646359 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:27.647347 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:27.647530 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:27.651859 (+ 4329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:56:27.651872 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:56:27.651892 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:56:27.651951 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:56:27.656222 (+ 4271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:27.656975 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:27.656984 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:27.656993 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:27.657252 (+ 259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:27.657638 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:27.657645 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:27.657651 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:27.658408 (+ 757us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:56:27.658716 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:27.659030 (+ 314us) server_negotiation.cc:299] Negotiation successful
0807 22:56:27.659504 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":682,"thread_start_us":417,"threads_started":1}
I20260807 22:56:27.661054 29907 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f7641d0236a24e8da5d450de8cf29d69 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:56:27.663429 29907 raft_consensus.cc:504] T 00000000000000000000000000000000 P f7641d0236a24e8da5d450de8cf29d69 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7641d0236a24e8da5d450de8cf29d69" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 34547 } }
I20260807 22:56:27.663959 29907 leader_election.cc:302] T 00000000000000000000000000000000 P f7641d0236a24e8da5d450de8cf29d69 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f7641d0236a24e8da5d450de8cf29d69; no voters:
I20260807 22:56:27.664484 29907 leader_election.cc:288] T 00000000000000000000000000000000 P f7641d0236a24e8da5d450de8cf29d69 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 22:56:27.664631 29916 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f7641d0236a24e8da5d450de8cf29d69 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:56:27.665410 29916 raft_consensus.cc:686] T 00000000000000000000000000000000 P f7641d0236a24e8da5d450de8cf29d69 [term 1 LEADER]: Becoming Leader. State: Replica: f7641d0236a24e8da5d450de8cf29d69, State: Running, Role: LEADER
I20260807 22:56:27.665990 29916 consensus_queue.cc:237] T 00000000000000000000000000000000 P f7641d0236a24e8da5d450de8cf29d69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7641d0236a24e8da5d450de8cf29d69" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 34547 } }
I20260807 22:56:27.666582 29907 sys_catalog.cc:564] T 00000000000000000000000000000000 P f7641d0236a24e8da5d450de8cf29d69 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 22:56:27.668901 29918 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7641d0236a24e8da5d450de8cf29d69 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f7641d0236a24e8da5d450de8cf29d69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7641d0236a24e8da5d450de8cf29d69" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 34547 } } }
I20260807 22:56:27.669407 29918 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7641d0236a24e8da5d450de8cf29d69 [sys.catalog]: This master's current role is: LEADER
I20260807 22:56:27.670195 29921 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7641d0236a24e8da5d450de8cf29d69 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f7641d0236a24e8da5d450de8cf29d69. Latest consensus state: current_term: 1 leader_uuid: "f7641d0236a24e8da5d450de8cf29d69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7641d0236a24e8da5d450de8cf29d69" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 34547 } } }
I20260807 22:56:27.670542 29921 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7641d0236a24e8da5d450de8cf29d69 [sys.catalog]: This master's current role is: LEADER
I20260807 22:56:27.671706 29937 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 22:56:27.676662 29937 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 22:56:27.685956 29937 catalog_manager.cc:1347] Generated new cluster ID: dd38d552dea64c90bcfb93321fe18721
I20260807 22:56:27.686094 29937 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 22:56:27.702489 29937 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 22:56:27.703936 29937 catalog_manager.cc:1504] Loading token signing keys...
I20260807 22:56:27.716285 29937 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f7641d0236a24e8da5d450de8cf29d69: Generated new TSK 0
I20260807 22:56:27.717254 29937 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.94.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.1.keytab.
Entry for principal kudu/127.25.94.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.1.keytab.
I20260807 22:56:27.770509 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786143361779964-25976-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786143361779964-25976-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786143361779964-25976-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786143361779964-25976-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.1:0
--local_ip_for_outbound_sockets=127.25.94.1
--webserver_interface=127.25.94.1
--webserver_port=0
--tserver_master_addrs=127.25.94.62:34547
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.1.keytab
--principal=kudu/127.25.94.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.20:44543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786143361779964-25976-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786143361779964-25976-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786143361779964-25976-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:56:28.123260 29952 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:56:28.123571 29952 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:56:28.123729 29952 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:56:28.131372 29952 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:56:28.131498 29952 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:56:28.131683 29952 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.1
I20260807 22:56:28.142143 29952 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:44543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786143361779964-25976-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786143361779964-25976-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.1.keytab
--principal=kudu/127.25.94.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786143361779964-25976-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.94.1
--webserver_port=0
--tserver_master_addrs=127.25.94.62:34547
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.1
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786143361779964-25976-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 22:56:28.143635 29952 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:56:28.145071 29952 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:56:28.161593 29959 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:56:28.161798 29958 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:56:28.163506 29961 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:56:28.165686 29952 server_base.cc:1034] running on GCE node
I20260807 22:56:28.167069 29952 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:56:28.168471 29952 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:56:28.169827 29952 hybrid_clock.cc:648] HybridClock initialized: now 1786143388169674 us; error 68 us; skew 500 ppm
I20260807 22:56:28.170706 29952 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 22:56:28 dist-test-slave-z8x9 krb5kdc[29767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143388, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:56:28.175550 29952 init.cc:377] Logged in from keytab as kudu/127.25.94.1@KRBTEST.COM (short username kudu)
I20260807 22:56:28.178099 29952 webserver.cc:460] Webserver started at http://127.25.94.1:32989/ using document root <none> and password file <none>
I20260807 22:56:28.178977 29952 fs_manager.cc:359] Metadata directory not provided
I20260807 22:56:28.179157 29952 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:56:28.179540 29952 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:56:28.182902 29952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786143361779964-25976-0/minicluster-data/ts-0/data/instance:
uuid: "818306db7b4e4b6da901de380ec3df56"
format_stamp: "Formatted at 2026-08-07 22:56:28 on dist-test-slave-z8x9"
server_key: "a92121396f972624c951b8f976a5baf5"
server_key_iv: "fa56387188eba5a67370f4fec14807e4"
server_key_version: "encryptionkey@0"
I20260807 22:56:28.183894 29952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786143361779964-25976-0/minicluster-data/ts-0/wal/instance:
uuid: "818306db7b4e4b6da901de380ec3df56"
format_stamp: "Formatted at 2026-08-07 22:56:28 on dist-test-slave-z8x9"
server_key: "a92121396f972624c951b8f976a5baf5"
server_key_iv: "fa56387188eba5a67370f4fec14807e4"
server_key_version: "encryptionkey@0"
I20260807 22:56:28.189960 29952 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.007s
I20260807 22:56:28.193912 29968 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:56:28.195488 29952 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 22:56:28.195672 29952 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786143361779964-25976-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786143361779964-25976-0/minicluster-data/ts-0/wal
uuid: "818306db7b4e4b6da901de380ec3df56"
format_stamp: "Formatted at 2026-08-07 22:56:28 on dist-test-slave-z8x9"
server_key: "a92121396f972624c951b8f976a5baf5"
server_key_iv: "fa56387188eba5a67370f4fec14807e4"
server_key_version: "encryptionkey@0"
I20260807 22:56:28.195891 29952 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786143361779964-25976-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786143361779964-25976-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786143361779964-25976-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:56:28.229974 29952 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:56:28.235267 29952 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:56:28.235611 29952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:56:28.236874 29952 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:56:28.238574 29952 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:56:28.238686 29952 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:56:28.238868 29952 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:56:28.238945 29952 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:56:28.278990 29952 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.1:36969
I20260807 22:56:28.279024 30137 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.1:36969 every 8 connection(s)
I20260807 22:56:28.280362 29952 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786143361779964-25976-0/minicluster-data/ts-0/data/info.pb
I20260807 22:56:28.286367 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 29952
I20260807 22:56:28.286634 25976 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786143361779964-25976-0/minicluster-data/ts-0/wal/instance
I20260807 22:56:28.287596 25976 external_mini_cluster.cc:1428] Setting key 830b0b1345bd0c0ee37b92d35c8f90df
Aug 07 22:56:28 dist-test-slave-z8x9 krb5kdc[29767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143388, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.1@KRBTEST.COM for kudu/127.25.94.62@KRBTEST.COM
I20260807 22:56:28.309789 30141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:28.283479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.1:44487 (local address 127.25.94.62:34547)
0807 22:56:28.284108 (+ 629us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:28.284113 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:28.285450 (+ 1337us) server_negotiation.cc:407] Connection header received
0807 22:56:28.287737 (+ 2287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:28.287743 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:28.287854 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:28.287983 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:56:28.289997 (+ 2014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:28.291365 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:28.294121 (+ 2756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:28.294933 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:28.295098 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:28.299700 (+ 4602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:56:28.299713 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:56:28.299721 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:56:28.299808 (+ 87us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:56:28.304737 (+ 4929us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:28.306039 (+ 1302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:28.306049 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:28.306054 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:28.306277 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:28.307033 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:28.307041 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:28.307047 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:28.307518 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:56:28.307708 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:28.308976 (+ 1268us) server_negotiation.cc:299] Negotiation successful
0807 22:56:28.309302 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":515,"thread_start_us":372,"threads_started":1}
WARNING: no policy specified for kudu/127.25.94.2@KRBTEST.COM; defaulting to no policy
I20260807 22:56:28.311363 30142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:28.284273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:34547 (local address 127.25.94.1:44487)
0807 22:56:28.285278 (+ 1005us) negotiation.cc:107] Waiting for socket to connect
0807 22:56:28.285324 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0807 22:56:28.287405 (+ 2081us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:56:28.288247 (+ 842us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:56:28.288256 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:56:28.289063 (+ 807us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:56:28.289792 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:28.289823 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:28.291569 (+ 1746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:28.291577 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:28.293938 (+ 2361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:28.293959 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:28.295220 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:28.295230 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:28.295587 (+ 357us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:28.296494 (+ 907us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:56:28.296531 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:56:28.299320 (+ 2789us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:56:28.304981 (+ 5661us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:56:28.304995 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:56:28.305020 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:56:28.305852 (+ 832us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:56:28.306474 (+ 622us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:56:28.306483 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:56:28.306489 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:56:28.306846 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:56:28.307694 (+ 848us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:56:28.307707 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:56:28.308414 (+ 707us) client_negotiation.cc:769] Sending connection context
0807 22:56:28.308821 (+ 407us) client_negotiation.cc:240] Negotiation successful
0807 22:56:28.309614 (+ 793us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":605,"thread_start_us":394,"threads_started":1}
I20260807 22:56:28.313302 30139 heartbeater.cc:347] Connected to a master server at 127.25.94.62:34547
I20260807 22:56:28.313678 30139 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.2@KRBTEST.COM" created.
I20260807 22:56:28.314623 30139 heartbeater.cc:511] Master 127.25.94.62:34547 requested a full tablet report, sending...
I20260807 22:56:28.316852 29842 ts_manager.cc:194] Registered new tserver with Master: 818306db7b4e4b6da901de380ec3df56 (127.25.94.1:36969)
I20260807 22:56:28.318491 29842 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.94.1@KRBTEST.COM'} at 127.25.94.1:44487
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.2.keytab.
Entry for principal kudu/127.25.94.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.2.keytab.
I20260807 22:56:28.346349 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786143361779964-25976-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786143361779964-25976-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786143361779964-25976-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786143361779964-25976-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.2:0
--local_ip_for_outbound_sockets=127.25.94.2
--webserver_interface=127.25.94.2
--webserver_port=0
--tserver_master_addrs=127.25.94.62:34547
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.2.keytab
--principal=kudu/127.25.94.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.20:44543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786143361779964-25976-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786143361779964-25976-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786143361779964-25976-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:56:28.690241 30147 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:56:28.690580 30147 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:56:28.690761 30147 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:56:28.698261 30147 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:56:28.698421 30147 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:56:28.698611 30147 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.2
I20260807 22:56:28.708519 30147 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:44543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786143361779964-25976-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786143361779964-25976-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.2.keytab
--principal=kudu/127.25.94.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786143361779964-25976-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.94.2
--webserver_port=0
--tserver_master_addrs=127.25.94.62:34547
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.2
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786143361779964-25976-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 22:56:28.709977 30147 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:56:28.711401 30147 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:56:28.724393 30153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:56:28.724370 30154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:56:28.724296 30156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:56:28.725232 30147 server_base.cc:1034] running on GCE node
I20260807 22:56:28.725883 30147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:56:28.726833 30147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:56:28.728068 30147 hybrid_clock.cc:648] HybridClock initialized: now 1786143388728010 us; error 68 us; skew 500 ppm
I20260807 22:56:28.728381 30147 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 22:56:28 dist-test-slave-z8x9 krb5kdc[29767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143388, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:56:28.731643 30147 init.cc:377] Logged in from keytab as kudu/127.25.94.2@KRBTEST.COM (short username kudu)
I20260807 22:56:28.733156 30147 webserver.cc:460] Webserver started at http://127.25.94.2:44689/ using document root <none> and password file <none>
I20260807 22:56:28.733925 30147 fs_manager.cc:359] Metadata directory not provided
I20260807 22:56:28.734076 30147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:56:28.734395 30147 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:56:28.737385 30147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786143361779964-25976-0/minicluster-data/ts-1/data/instance:
uuid: "cbbc35e840474d15835d95cd45c40972"
format_stamp: "Formatted at 2026-08-07 22:56:28 on dist-test-slave-z8x9"
server_key: "4e97037fbeeeed51b1832f70ee044a01"
server_key_iv: "58fc8d641ddc3d3a6d11f808c46d659d"
server_key_version: "encryptionkey@0"
I20260807 22:56:28.738186 30147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786143361779964-25976-0/minicluster-data/ts-1/wal/instance:
uuid: "cbbc35e840474d15835d95cd45c40972"
format_stamp: "Formatted at 2026-08-07 22:56:28 on dist-test-slave-z8x9"
server_key: "4e97037fbeeeed51b1832f70ee044a01"
server_key_iv: "58fc8d641ddc3d3a6d11f808c46d659d"
server_key_version: "encryptionkey@0"
I20260807 22:56:28.743404 30147 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.005s
I20260807 22:56:28.746956 30164 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:56:28.748413 30147 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260807 22:56:28.748608 30147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786143361779964-25976-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786143361779964-25976-0/minicluster-data/ts-1/wal
uuid: "cbbc35e840474d15835d95cd45c40972"
format_stamp: "Formatted at 2026-08-07 22:56:28 on dist-test-slave-z8x9"
server_key: "4e97037fbeeeed51b1832f70ee044a01"
server_key_iv: "58fc8d641ddc3d3a6d11f808c46d659d"
server_key_version: "encryptionkey@0"
I20260807 22:56:28.748827 30147 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786143361779964-25976-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786143361779964-25976-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786143361779964-25976-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:56:28.764954 30147 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:56:28.769686 30147 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:56:28.769942 30147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:56:28.770991 30147 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:56:28.772289 30147 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:56:28.772364 30147 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:56:28.772450 30147 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:56:28.772516 30147 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:56:28.807174 30147 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.2:45405
I20260807 22:56:28.807224 30332 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.2:45405 every 8 connection(s)
I20260807 22:56:28.808483 30147 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786143361779964-25976-0/minicluster-data/ts-1/data/info.pb
I20260807 22:56:28.810982 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 30147
I20260807 22:56:28.811190 25976 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786143361779964-25976-0/minicluster-data/ts-1/wal/instance
I20260807 22:56:28.811937 25976 external_mini_cluster.cc:1428] Setting key 64bd295594c4c77b9ba9055ac42e602b
Aug 07 22:56:28 dist-test-slave-z8x9 krb5kdc[29767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143388, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.2@KRBTEST.COM for kudu/127.25.94.62@KRBTEST.COM
I20260807 22:56:28.836298 30338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:28.811648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:34547 (local address 127.25.94.2:42021)
0807 22:56:28.812886 (+ 1238us) negotiation.cc:107] Waiting for socket to connect
0807 22:56:28.812944 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0807 22:56:28.815238 (+ 2294us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:56:28.816008 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:56:28.816023 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:56:28.816874 (+ 851us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:56:28.817749 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:28.817789 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:28.818885 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:28.818894 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:28.821104 (+ 2210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:28.821122 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:28.822113 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:28.822118 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:28.822332 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:28.823104 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:56:28.823127 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:56:28.825697 (+ 2570us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:56:28.830132 (+ 4435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:56:28.830145 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:56:28.830168 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:56:28.830947 (+ 779us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:56:28.831492 (+ 545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:56:28.831500 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:56:28.831506 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:56:28.831807 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:56:28.832790 (+ 983us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:56:28.832800 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:56:28.833373 (+ 573us) client_negotiation.cc:769] Sending connection context
0807 22:56:28.833722 (+ 349us) client_negotiation.cc:240] Negotiation successful
0807 22:56:28.834397 (+ 675us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":647,"thread_start_us":454,"threads_started":1}
WARNING: no policy specified for kudu/127.25.94.3@KRBTEST.COM; defaulting to no policy
I20260807 22:56:28.837618 30335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:28.811112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.2:42021 (local address 127.25.94.62:34547)
0807 22:56:28.812928 (+ 1816us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:28.812936 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:28.813056 (+ 120us) server_negotiation.cc:407] Connection header received
0807 22:56:28.815532 (+ 2476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:28.815536 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:28.815602 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:28.815701 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:56:28.817960 (+ 2259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:28.818689 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:28.821293 (+ 2604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:28.821887 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:28.822029 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:28.825991 (+ 3962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:56:28.826001 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:56:28.826009 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:56:28.826056 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:56:28.829931 (+ 3875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:28.831120 (+ 1189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:28.831129 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:28.831135 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:28.831334 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:28.832007 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:28.832015 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:28.832020 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:28.832518 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:56:28.832687 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:28.836769 (+ 4082us) server_negotiation.cc:299] Negotiation successful
0807 22:56:28.837092 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1688,"thread_start_us":1581,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.3@KRBTEST.COM" created.
I20260807 22:56:28.840058 30333 heartbeater.cc:347] Connected to a master server at 127.25.94.62:34547
I20260807 22:56:28.840410 30333 heartbeater.cc:464] Registering TS with master...
I20260807 22:56:28.841426 30333 heartbeater.cc:511] Master 127.25.94.62:34547 requested a full tablet report, sending...
I20260807 22:56:28.843163 29842 ts_manager.cc:194] Registered new tserver with Master: cbbc35e840474d15835d95cd45c40972 (127.25.94.2:45405)
I20260807 22:56:28.843995 29842 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.94.2@KRBTEST.COM'} at 127.25.94.2:42021
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.3.keytab.
Entry for principal kudu/127.25.94.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.3.keytab.
I20260807 22:56:28.866644 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786143361779964-25976-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786143361779964-25976-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786143361779964-25976-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786143361779964-25976-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.3:0
--local_ip_for_outbound_sockets=127.25.94.3
--webserver_interface=127.25.94.3
--webserver_port=0
--tserver_master_addrs=127.25.94.62:34547
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.3.keytab
--principal=kudu/127.25.94.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.20:44543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786143361779964-25976-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786143361779964-25976-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786143361779964-25976-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:56:29.206562 30346 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:56:29.206892 30346 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:56:29.207067 30346 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:56:29.214701 30346 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:56:29.214885 30346 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:56:29.215077 30346 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.3
I20260807 22:56:29.225288 30346 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:44543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786143361779964-25976-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786143361779964-25976-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.3.keytab
--principal=kudu/127.25.94.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786143361779964-25976-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.94.3
--webserver_port=0
--tserver_master_addrs=127.25.94.62:34547
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.3
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786143361779964-25976-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 22:56:29.226816 30346 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:56:29.228196 30346 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:56:29.242027 30357 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:56:29.242097 30359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:56:29.242380 30363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:56:29.243158 30346 server_base.cc:1034] running on GCE node
I20260807 22:56:29.243870 30346 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:56:29.244839 30346 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:56:29.246071 30346 hybrid_clock.cc:648] HybridClock initialized: now 1786143389246025 us; error 57 us; skew 500 ppm
I20260807 22:56:29.246413 30346 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 22:56:29 dist-test-slave-z8x9 krb5kdc[29767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143389, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:56:29.249851 30346 init.cc:377] Logged in from keytab as kudu/127.25.94.3@KRBTEST.COM (short username kudu)
I20260807 22:56:29.251272 30346 webserver.cc:460] Webserver started at http://127.25.94.3:35563/ using document root <none> and password file <none>
I20260807 22:56:29.251977 30346 fs_manager.cc:359] Metadata directory not provided
I20260807 22:56:29.252175 30346 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:56:29.252529 30346 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:56:29.255527 30346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786143361779964-25976-0/minicluster-data/ts-2/data/instance:
uuid: "b6dde2801a63447cb1f3d7ddcafc5a09"
format_stamp: "Formatted at 2026-08-07 22:56:29 on dist-test-slave-z8x9"
server_key: "4a17e6130f712ec218ca12176438f88f"
server_key_iv: "d2073821fb875c0e711b51f73c9378de"
server_key_version: "encryptionkey@0"
I20260807 22:56:29.256338 30346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786143361779964-25976-0/minicluster-data/ts-2/wal/instance:
uuid: "b6dde2801a63447cb1f3d7ddcafc5a09"
format_stamp: "Formatted at 2026-08-07 22:56:29 on dist-test-slave-z8x9"
server_key: "4a17e6130f712ec218ca12176438f88f"
server_key_iv: "d2073821fb875c0e711b51f73c9378de"
server_key_version: "encryptionkey@0"
I20260807 22:56:29.261724 30346 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260807 22:56:29.265125 30372 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:56:29.266525 30346 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 22:56:29.266705 30346 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786143361779964-25976-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786143361779964-25976-0/minicluster-data/ts-2/wal
uuid: "b6dde2801a63447cb1f3d7ddcafc5a09"
format_stamp: "Formatted at 2026-08-07 22:56:29 on dist-test-slave-z8x9"
server_key: "4a17e6130f712ec218ca12176438f88f"
server_key_iv: "d2073821fb875c0e711b51f73c9378de"
server_key_version: "encryptionkey@0"
I20260807 22:56:29.266908 30346 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786143361779964-25976-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786143361779964-25976-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786143361779964-25976-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:56:29.287599 30346 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:56:29.292061 30346 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:56:29.292387 30346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:56:29.293493 30346 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:56:29.294929 30346 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:56:29.295058 30346 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:56:29.295212 30346 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:56:29.295287 30346 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:56:29.321552 30139 heartbeater.cc:503] Master 127.25.94.62:34547 was elected leader, sending a full tablet report...
I20260807 22:56:29.330853 30346 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.3:35493
I20260807 22:56:29.330893 30553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.3:35493 every 8 connection(s)
I20260807 22:56:29.332185 30346 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786143361779964-25976-0/minicluster-data/ts-2/data/info.pb
I20260807 22:56:29.338768 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 30346
I20260807 22:56:29.338989 25976 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786143361779964-25976-0/minicluster-data/ts-2/wal/instance
I20260807 22:56:29.339416 25976 external_mini_cluster.cc:1428] Setting key 603dcc39255b04e832e0383d4e12d2a5
Aug 07 22:56:29 dist-test-slave-z8x9 krb5kdc[29767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143389, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.3@KRBTEST.COM for kudu/127.25.94.62@KRBTEST.COM
I20260807 22:56:29.353431 30335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:29.335027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.3:36257 (local address 127.25.94.62:34547)
0807 22:56:29.335237 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:29.335242 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:29.336384 (+ 1142us) server_negotiation.cc:407] Connection header received
0807 22:56:29.338357 (+ 1973us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:29.338363 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:29.338475 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:29.338593 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:56:29.340408 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:29.341384 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:29.343069 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:29.343474 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:29.343616 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:29.347177 (+ 3561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:56:29.347182 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:56:29.347186 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:56:29.347219 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:56:29.350300 (+ 3081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:29.351043 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:29.351051 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:29.351057 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:29.351186 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:29.351623 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:29.351629 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:29.351632 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:29.351911 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:56:29.352065 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:29.352843 (+ 778us) server_negotiation.cc:299] Negotiation successful
0807 22:56:29.353082 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260807 22:56:29.354550 30558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:29.335429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:34547 (local address 127.25.94.3:36257)
0807 22:56:29.336172 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0807 22:56:29.336212 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0807 22:56:29.338071 (+ 1859us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:56:29.338833 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:56:29.338840 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:56:29.339507 (+ 667us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:56:29.340230 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:29.340257 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:29.341525 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:29.341531 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:29.342908 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:29.342920 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:29.343710 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:29.343718 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:29.343954 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:29.344617 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:56:29.344644 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:56:29.346944 (+ 2300us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:56:29.350447 (+ 3503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:56:29.350456 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:56:29.350472 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:56:29.350918 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:56:29.351305 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:56:29.351310 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:56:29.351314 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:56:29.351506 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:56:29.352039 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:56:29.352047 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:56:29.352422 (+ 375us) client_negotiation.cc:769] Sending connection context
0807 22:56:29.352755 (+ 333us) client_negotiation.cc:240] Negotiation successful
0807 22:56:29.353285 (+ 530us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":306,"threads_started":1}
I20260807 22:56:29.356003 30556 heartbeater.cc:347] Connected to a master server at 127.25.94.62:34547
I20260807 22:56:29.356315 30556 heartbeater.cc:464] Registering TS with master...
I20260807 22:56:29.357067 30556 heartbeater.cc:511] Master 127.25.94.62:34547 requested a full tablet report, sending...
I20260807 22:56:29.358500 29842 ts_manager.cc:194] Registered new tserver with Master: b6dde2801a63447cb1f3d7ddcafc5a09 (127.25.94.3:35493)
I20260807 22:56:29.359292 29842 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.94.3@KRBTEST.COM'} at 127.25.94.3:36257
I20260807 22:56:29.367471 25976 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260807 22:56:29.404548 30335 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:56:29.402060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60802 (local address 127.25.94.62:34547)
0807 22:56:29.402269 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:29.402275 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:29.402750 (+ 475us) server_negotiation.cc:407] Connection header received
0807 22:56:29.402907 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:29.402912 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:29.402981 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:29.403078 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:56:29.404312 (+ 1234us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60802: BlockingRecv error: recv got EOF from 127.0.0.1:60802 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260807 22:56:29.407590 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 29952
W20260807 22:56:29.410434 25976 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 29952: Not authorized: Client connection negotiation failed: client connection to 127.25.94.1:36969: server requires authentication, but client does not have Kerberos credentials available
I20260807 22:56:29.410693 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 29952
I20260807 22:56:29.435540 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 30147
W20260807 22:56:29.438441 25976 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 30147: Not authorized: Client connection negotiation failed: client connection to 127.25.94.2:45405: server requires authentication, but client does not have Kerberos credentials available
I20260807 22:56:29.438606 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 30147
I20260807 22:56:29.460947 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 30346
W20260807 22:56:29.464066 25976 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 30346: Not authorized: Client connection negotiation failed: client connection to 127.25.94.3:35493: server requires authentication, but client does not have Kerberos credentials available
I20260807 22:56:29.464221 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 30346
I20260807 22:56:29.485987 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 29793
I20260807 22:56:29.487248 29901 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 22:56:29.594553 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 29793
2026-08-07T22:56:29Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (4015 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786143361779964-25976-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 22:56:29 dist-test-slave-z8x9 krb5kdc[30584](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 22:56:29 dist-test-slave-z8x9 krb5kdc[30584](info): set up 2 sockets
Aug 07 22:56:29 dist-test-slave-z8x9 krb5kdc[30584](info): commencing operation
krb5kdc: starting...
W20260807 22:56:30.951644 25976 mini_kdc.cc:121] Time spent starting KDC: real 1.305s user 0.000s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786143361779964-25976-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786143361779964-25976-0/krb5kdc/test-user.keytab.
Aug 07 22:56:31 dist-test-slave-z8x9 krb5kdc[30584](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143391, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T22:56:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T22:56:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.94.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.62.keytab.
Entry for principal kudu/127.25.94.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.62.keytab.
I20260807 22:56:31.186902 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786143361779964-25976-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786143361779964-25976-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786143361779964-25976-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.62:38503
--webserver_interface=127.25.94.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.62.keytab
--principal=kudu/127.25.94.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.20:35257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.94.62:38503
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786143361779964-25976-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786143361779964-25976-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786143361779964-25976-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:56:31.528005 30605 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:56:31.528424 30605 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:56:31.528604 30605 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:56:31.536679 30605 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 22:56:31.536842 30605 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:56:31.536939 30605 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:56:31.537065 30605 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 22:56:31.537185 30605 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 22:56:31.548496 30605 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:35257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786143361779964-25976-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786143361779964-25976-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.94.62:38503
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.62.keytab
--principal=kudu/127.25.94.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.62:38503
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.94.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786143361779964-25976-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 22:56:31.550001 30605 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:56:31.551394 30605 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:56:31.564071 30614 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:56:31.564095 30619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:56:31.564025 30615 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:56:31.565429 30605 server_base.cc:1034] running on GCE node
I20260807 22:56:31.566660 30605 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:56:31.568141 30605 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:56:31.569377 30605 hybrid_clock.cc:648] HybridClock initialized: now 1786143391569338 us; error 60 us; skew 500 ppm
I20260807 22:56:31.569777 30605 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 22:56:31 dist-test-slave-z8x9 krb5kdc[30584](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143391, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:56:31.578148 30605 init.cc:377] Logged in from keytab as kudu/127.25.94.62@KRBTEST.COM (short username kudu)
I20260807 22:56:31.580108 30605 webserver.cc:460] Webserver started at http://127.25.94.62:33381/ using document root <none> and password file <none>
I20260807 22:56:31.580821 30605 fs_manager.cc:359] Metadata directory not provided
I20260807 22:56:31.580930 30605 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:56:31.581292 30605 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:56:31.584553 30605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786143361779964-25976-0/minicluster-data/master-0/data/instance:
uuid: "a822035d675543ba9c57892e11fb3b92"
format_stamp: "Formatted at 2026-08-07 22:56:31 on dist-test-slave-z8x9"
server_key: "0d3e1f68a9aa41d0364842f0f62e4a85"
server_key_iv: "48576fd14d71aa3b0e1abb201f818e30"
server_key_version: "encryptionkey@0"
I20260807 22:56:31.586323 30605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786143361779964-25976-0/minicluster-data/master-0/wal/instance:
uuid: "a822035d675543ba9c57892e11fb3b92"
format_stamp: "Formatted at 2026-08-07 22:56:31 on dist-test-slave-z8x9"
server_key: "0d3e1f68a9aa41d0364842f0f62e4a85"
server_key_iv: "48576fd14d71aa3b0e1abb201f818e30"
server_key_version: "encryptionkey@0"
I20260807 22:56:31.592077 30605 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260807 22:56:31.595732 30635 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:56:31.597252 30605 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260807 22:56:31.597419 30605 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786143361779964-25976-0/minicluster-data/master-0/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786143361779964-25976-0/minicluster-data/master-0/wal
uuid: "a822035d675543ba9c57892e11fb3b92"
format_stamp: "Formatted at 2026-08-07 22:56:31 on dist-test-slave-z8x9"
server_key: "0d3e1f68a9aa41d0364842f0f62e4a85"
server_key_iv: "48576fd14d71aa3b0e1abb201f818e30"
server_key_version: "encryptionkey@0"
I20260807 22:56:31.597606 30605 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786143361779964-25976-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786143361779964-25976-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786143361779964-25976-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:56:31.623274 30605 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:56:31.627689 30605 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:56:31.627951 30605 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:56:31.648252 30605 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.62:38503
I20260807 22:56:31.648372 30711 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.62:38503 every 8 connection(s)
I20260807 22:56:31.649835 30605 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
I20260807 22:56:31.650693 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 30605
I20260807 22:56:31.650946 25976 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786143361779964-25976-0/minicluster-data/master-0/wal/instance
I20260807 22:56:31.651793 25976 external_mini_cluster.cc:1428] Setting key 2714354283806bfa1c6268dadc0460af
I20260807 22:56:31.655139 30713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 07 22:56:31 dist-test-slave-z8x9 krb5kdc[30584](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143391, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.62@KRBTEST.COM
I20260807 22:56:31.669203 30713 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a822035d675543ba9c57892e11fb3b92: Bootstrap starting.
I20260807 22:56:31.673609 30713 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a822035d675543ba9c57892e11fb3b92: Neither blocks nor log segments found. Creating new log.
I20260807 22:56:31.674813 30713 log.cc:824] T 00000000000000000000000000000000 P a822035d675543ba9c57892e11fb3b92: Log is configured to *not* fsync() on all Append() calls
I20260807 22:56:31.678213 30722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:31.653859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42342 (local address 127.25.94.62:38503)
0807 22:56:31.654890 (+ 1031us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:31.654902 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:31.654944 (+ 42us) server_negotiation.cc:407] Connection header received
0807 22:56:31.656693 (+ 1749us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:31.656716 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:31.657248 (+ 532us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:31.657588 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:56:31.659701 (+ 2113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:31.661305 (+ 1604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:31.663109 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:31.663788 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:31.663918 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:31.669111 (+ 5193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:56:31.669122 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:56:31.669142 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:56:31.669243 (+ 101us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:56:31.673337 (+ 4094us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:31.674144 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:31.674152 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:31.674160 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:31.674379 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:31.674763 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:31.674768 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:31.674771 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:31.675589 (+ 818us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:56:31.675820 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:31.676413 (+ 593us) server_negotiation.cc:299] Negotiation successful
0807 22:56:31.676864 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":647,"thread_start_us":414,"threads_started":1}
I20260807 22:56:31.678388 30713 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a822035d675543ba9c57892e11fb3b92: No bootstrap required, opened a new log
I20260807 22:56:31.682442 25976 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 22:56:31.686796 30713 raft_consensus.cc:348] T 00000000000000000000000000000000 P a822035d675543ba9c57892e11fb3b92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a822035d675543ba9c57892e11fb3b92" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 38503 } }
I20260807 22:56:31.687295 30713 raft_consensus.cc:374] T 00000000000000000000000000000000 P a822035d675543ba9c57892e11fb3b92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:56:31.687451 30713 raft_consensus.cc:729] T 00000000000000000000000000000000 P a822035d675543ba9c57892e11fb3b92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a822035d675543ba9c57892e11fb3b92, State: Initialized, Role: FOLLOWER
I20260807 22:56:31.688150 30713 consensus_queue.cc:260] T 00000000000000000000000000000000 P a822035d675543ba9c57892e11fb3b92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a822035d675543ba9c57892e11fb3b92" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 38503 } }
I20260807 22:56:31.688421 30713 raft_consensus.cc:388] T 00000000000000000000000000000000 P a822035d675543ba9c57892e11fb3b92 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 22:56:31.688561 30713 raft_consensus.cc:482] T 00000000000000000000000000000000 P a822035d675543ba9c57892e11fb3b92 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 22:56:31.688751 30713 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a822035d675543ba9c57892e11fb3b92 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:56:31.691076 30713 raft_consensus.cc:504] T 00000000000000000000000000000000 P a822035d675543ba9c57892e11fb3b92 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a822035d675543ba9c57892e11fb3b92" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 38503 } }
I20260807 22:56:31.691633 30713 leader_election.cc:302] T 00000000000000000000000000000000 P a822035d675543ba9c57892e11fb3b92 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a822035d675543ba9c57892e11fb3b92; no voters:
I20260807 22:56:31.692363 30713 leader_election.cc:288] T 00000000000000000000000000000000 P a822035d675543ba9c57892e11fb3b92 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 22:56:31.692472 30729 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a822035d675543ba9c57892e11fb3b92 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:56:31.693730 30729 raft_consensus.cc:686] T 00000000000000000000000000000000 P a822035d675543ba9c57892e11fb3b92 [term 1 LEADER]: Becoming Leader. State: Replica: a822035d675543ba9c57892e11fb3b92, State: Running, Role: LEADER
I20260807 22:56:31.694346 30729 consensus_queue.cc:237] T 00000000000000000000000000000000 P a822035d675543ba9c57892e11fb3b92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a822035d675543ba9c57892e11fb3b92" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 38503 } }
I20260807 22:56:31.694952 30713 sys_catalog.cc:564] T 00000000000000000000000000000000 P a822035d675543ba9c57892e11fb3b92 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 22:56:31.697738 30730 sys_catalog.cc:455] T 00000000000000000000000000000000 P a822035d675543ba9c57892e11fb3b92 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a822035d675543ba9c57892e11fb3b92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a822035d675543ba9c57892e11fb3b92" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 38503 } } }
I20260807 22:56:31.698277 30730 sys_catalog.cc:458] T 00000000000000000000000000000000 P a822035d675543ba9c57892e11fb3b92 [sys.catalog]: This master's current role is: LEADER
I20260807 22:56:31.699812 30731 sys_catalog.cc:455] T 00000000000000000000000000000000 P a822035d675543ba9c57892e11fb3b92 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a822035d675543ba9c57892e11fb3b92. Latest consensus state: current_term: 1 leader_uuid: "a822035d675543ba9c57892e11fb3b92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a822035d675543ba9c57892e11fb3b92" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 38503 } } }
I20260807 22:56:31.700258 30731 sys_catalog.cc:458] T 00000000000000000000000000000000 P a822035d675543ba9c57892e11fb3b92 [sys.catalog]: This master's current role is: LEADER
I20260807 22:56:31.701633 30737 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 22:56:31.708906 30737 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 22:56:31.722532 30737 catalog_manager.cc:1347] Generated new cluster ID: 0edc51188c314c3cb48dea064a4a220c
I20260807 22:56:31.722695 30737 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 22:56:31.736292 30737 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 22:56:31.738332 30737 catalog_manager.cc:1504] Loading token signing keys...
I20260807 22:56:31.740046 30722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:31.731255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42350 (local address 127.25.94.62:38503)
0807 22:56:31.731451 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:31.731458 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:31.732145 (+ 687us) server_negotiation.cc:407] Connection header received
0807 22:56:31.732333 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:31.732340 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:31.732435 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:31.732545 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:56:31.733978 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:31.734982 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:31.736579 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:31.737054 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:31.737262 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:31.738419 (+ 1157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:56:31.738426 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:56:31.738434 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 22:56:31.739122 (+ 688us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:56:31.739168 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 22:56:31.739197 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:56:31.739364 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:31.739490 (+ 126us) server_negotiation.cc:299] Negotiation successful
0807 22:56:31.739626 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":59}
I20260807 22:56:31.759265 30737 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a822035d675543ba9c57892e11fb3b92: Generated new TSK 0
I20260807 22:56:31.760174 30737 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 22:56:31.768628 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 30605
I20260807 22:56:31.769829 30704 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 22:56:31.858888 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 30605
2026-08-07T22:56:31Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (2256 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786143361779964-25976-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 22:56:31 dist-test-slave-z8x9 krb5kdc[30775](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 22:56:31 dist-test-slave-z8x9 krb5kdc[30775](info): set up 2 sockets
Aug 07 22:56:31 dist-test-slave-z8x9 krb5kdc[30775](info): commencing operation
krb5kdc: starting...
W20260807 22:56:33.197806 25976 mini_kdc.cc:121] Time spent starting KDC: real 1.295s user 0.006s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786143361779964-25976-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786143361779964-25976-0/krb5kdc/test-user.keytab.
Aug 07 22:56:33 dist-test-slave-z8x9 krb5kdc[30775](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143393, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T22:56:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T22:56:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.94.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.62.keytab.
Entry for principal kudu/127.25.94.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.62.keytab.
I20260807 22:56:33.433030 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786143361779964-25976-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786143361779964-25976-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786143361779964-25976-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.62:37089
--webserver_interface=127.25.94.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.62.keytab
--principal=kudu/127.25.94.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.20:42763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.94.62:37089
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786143361779964-25976-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786143361779964-25976-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786143361779964-25976-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:56:33.775110 30796 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:56:33.775419 30796 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:56:33.775517 30796 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:56:33.782763 30796 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 22:56:33.782883 30796 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:56:33.782933 30796 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:56:33.782989 30796 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 22:56:33.783034 30796 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 22:56:33.793306 30796 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:42763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786143361779964-25976-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786143361779964-25976-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.94.62:37089
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.62.keytab
--principal=kudu/127.25.94.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.62:37089
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.94.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786143361779964-25976-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 22:56:33.794620 30796 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:56:33.795904 30796 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:56:33.807562 30812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:56:33.807595 30806 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:56:33.807701 30805 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:56:33.808650 30796 server_base.cc:1034] running on GCE node
I20260807 22:56:33.809528 30796 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:56:33.810889 30796 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:56:33.812103 30796 hybrid_clock.cc:648] HybridClock initialized: now 1786143393812053 us; error 31 us; skew 500 ppm
I20260807 22:56:33.812458 30796 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 22:56:33 dist-test-slave-z8x9 krb5kdc[30775](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143393, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:56:33.815974 30796 init.cc:377] Logged in from keytab as kudu/127.25.94.62@KRBTEST.COM (short username kudu)
I20260807 22:56:33.817577 30796 webserver.cc:460] Webserver started at http://127.25.94.62:39347/ using document root <none> and password file <none>
I20260807 22:56:33.818236 30796 fs_manager.cc:359] Metadata directory not provided
I20260807 22:56:33.818346 30796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:56:33.818608 30796 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:56:33.821826 30796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786143361779964-25976-0/minicluster-data/master-0/data/instance:
uuid: "5677fe2d33dc4df1a6ee417e6107fd7b"
format_stamp: "Formatted at 2026-08-07 22:56:33 on dist-test-slave-z8x9"
server_key: "e0a79af1ed9c81be2d539c9a00dc10f4"
server_key_iv: "ecd2c43ef1ea65b84c9819adc5a242f2"
server_key_version: "encryptionkey@0"
I20260807 22:56:33.822578 30796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786143361779964-25976-0/minicluster-data/master-0/wal/instance:
uuid: "5677fe2d33dc4df1a6ee417e6107fd7b"
format_stamp: "Formatted at 2026-08-07 22:56:33 on dist-test-slave-z8x9"
server_key: "e0a79af1ed9c81be2d539c9a00dc10f4"
server_key_iv: "ecd2c43ef1ea65b84c9819adc5a242f2"
server_key_version: "encryptionkey@0"
I20260807 22:56:33.827982 30796 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260807 22:56:33.831414 30821 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:56:33.832737 30796 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260807 22:56:33.832942 30796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786143361779964-25976-0/minicluster-data/master-0/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786143361779964-25976-0/minicluster-data/master-0/wal
uuid: "5677fe2d33dc4df1a6ee417e6107fd7b"
format_stamp: "Formatted at 2026-08-07 22:56:33 on dist-test-slave-z8x9"
server_key: "e0a79af1ed9c81be2d539c9a00dc10f4"
server_key_iv: "ecd2c43ef1ea65b84c9819adc5a242f2"
server_key_version: "encryptionkey@0"
I20260807 22:56:33.833144 30796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786143361779964-25976-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786143361779964-25976-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786143361779964-25976-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:56:33.845886 30796 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:56:33.850718 30796 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:56:33.851045 30796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:56:33.870888 30796 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.62:37089
I20260807 22:56:33.870918 30900 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.62:37089 every 8 connection(s)
I20260807 22:56:33.872359 30796 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
I20260807 22:56:33.874257 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 30796
I20260807 22:56:33.874486 25976 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786143361779964-25976-0/minicluster-data/master-0/wal/instance
I20260807 22:56:33.875339 25976 external_mini_cluster.cc:1428] Setting key ca8db0dbc7b6ab940779b6b02af63ade
I20260807 22:56:33.877280 30901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 07 22:56:33 dist-test-slave-z8x9 krb5kdc[30775](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143393, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.62@KRBTEST.COM
I20260807 22:56:33.892129 30901 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5677fe2d33dc4df1a6ee417e6107fd7b: Bootstrap starting.
I20260807 22:56:33.895829 30901 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5677fe2d33dc4df1a6ee417e6107fd7b: Neither blocks nor log segments found. Creating new log.
I20260807 22:56:33.897241 30901 log.cc:824] T 00000000000000000000000000000000 P 5677fe2d33dc4df1a6ee417e6107fd7b: Log is configured to *not* fsync() on all Append() calls
I20260807 22:56:33.899941 30906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:33.877653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60874 (local address 127.25.94.62:37089)
0807 22:56:33.878455 (+ 802us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:33.878467 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:33.878508 (+ 41us) server_negotiation.cc:407] Connection header received
0807 22:56:33.880047 (+ 1539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:33.880067 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:33.880473 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:33.880810 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:56:33.882593 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:33.883924 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:33.885848 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:33.886504 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:33.886639 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:33.892103 (+ 5464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:56:33.892115 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:56:33.892133 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:56:33.892173 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:56:33.895521 (+ 3348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:33.896462 (+ 941us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:33.896470 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:33.896476 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:33.896649 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:33.897099 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:33.897102 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:33.897104 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:33.897834 (+ 730us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:56:33.898185 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:33.898600 (+ 415us) server_negotiation.cc:299] Negotiation successful
0807 22:56:33.898990 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":583,"thread_start_us":363,"threads_started":1}
I20260807 22:56:33.900480 30901 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5677fe2d33dc4df1a6ee417e6107fd7b: No bootstrap required, opened a new log
I20260807 22:56:33.908718 30901 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5677fe2d33dc4df1a6ee417e6107fd7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5677fe2d33dc4df1a6ee417e6107fd7b" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 37089 } }
I20260807 22:56:33.909240 30901 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5677fe2d33dc4df1a6ee417e6107fd7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:56:33.909382 30901 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5677fe2d33dc4df1a6ee417e6107fd7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5677fe2d33dc4df1a6ee417e6107fd7b, State: Initialized, Role: FOLLOWER
I20260807 22:56:33.910063 30901 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5677fe2d33dc4df1a6ee417e6107fd7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5677fe2d33dc4df1a6ee417e6107fd7b" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 37089 } }
I20260807 22:56:33.910306 30901 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5677fe2d33dc4df1a6ee417e6107fd7b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 22:56:33.910449 30901 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5677fe2d33dc4df1a6ee417e6107fd7b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 22:56:33.910655 30901 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5677fe2d33dc4df1a6ee417e6107fd7b [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:56:33.913122 30901 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5677fe2d33dc4df1a6ee417e6107fd7b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5677fe2d33dc4df1a6ee417e6107fd7b" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 37089 } }
I20260807 22:56:33.913789 30901 leader_election.cc:302] T 00000000000000000000000000000000 P 5677fe2d33dc4df1a6ee417e6107fd7b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5677fe2d33dc4df1a6ee417e6107fd7b; no voters:
I20260807 22:56:33.914423 30901 leader_election.cc:288] T 00000000000000000000000000000000 P 5677fe2d33dc4df1a6ee417e6107fd7b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 22:56:33.914569 30910 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5677fe2d33dc4df1a6ee417e6107fd7b [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:56:33.915311 30910 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5677fe2d33dc4df1a6ee417e6107fd7b [term 1 LEADER]: Becoming Leader. State: Replica: 5677fe2d33dc4df1a6ee417e6107fd7b, State: Running, Role: LEADER
I20260807 22:56:33.915867 30910 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5677fe2d33dc4df1a6ee417e6107fd7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5677fe2d33dc4df1a6ee417e6107fd7b" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 37089 } }
I20260807 22:56:33.916771 30901 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5677fe2d33dc4df1a6ee417e6107fd7b [sys.catalog]: configured and running, proceeding with master startup.
I20260807 22:56:33.919020 30914 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5677fe2d33dc4df1a6ee417e6107fd7b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5677fe2d33dc4df1a6ee417e6107fd7b. Latest consensus state: current_term: 1 leader_uuid: "5677fe2d33dc4df1a6ee417e6107fd7b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5677fe2d33dc4df1a6ee417e6107fd7b" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 37089 } } }
I20260807 22:56:33.919198 30912 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5677fe2d33dc4df1a6ee417e6107fd7b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5677fe2d33dc4df1a6ee417e6107fd7b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5677fe2d33dc4df1a6ee417e6107fd7b" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 37089 } } }
I20260807 22:56:33.919487 30912 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5677fe2d33dc4df1a6ee417e6107fd7b [sys.catalog]: This master's current role is: LEADER
I20260807 22:56:33.919487 30914 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5677fe2d33dc4df1a6ee417e6107fd7b [sys.catalog]: This master's current role is: LEADER
I20260807 22:56:33.922238 30924 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 22:56:33.927814 30924 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 22:56:33.936355 30924 catalog_manager.cc:1347] Generated new cluster ID: 5b2cc51e3fc64179b74f26a461ebb0ae
I20260807 22:56:33.936501 30924 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 22:56:33.948828 30924 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 22:56:33.950173 30924 catalog_manager.cc:1504] Loading token signing keys...
I20260807 22:56:33.962558 30924 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5677fe2d33dc4df1a6ee417e6107fd7b: Generated new TSK 0
I20260807 22:56:33.963529 30924 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.94.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.1.keytab.
Entry for principal kudu/127.25.94.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.1.keytab.
I20260807 22:56:34.018712 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786143361779964-25976-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786143361779964-25976-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786143361779964-25976-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786143361779964-25976-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.1:0
--local_ip_for_outbound_sockets=127.25.94.1
--webserver_interface=127.25.94.1
--webserver_port=0
--tserver_master_addrs=127.25.94.62:37089
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.1.keytab
--principal=kudu/127.25.94.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.20:42763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786143361779964-25976-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786143361779964-25976-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786143361779964-25976-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:56:34.372314 30951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:56:34.372614 30951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:56:34.372779 30951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:56:34.380113 30951 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:56:34.380230 30951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:56:34.380451 30951 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.1
I20260807 22:56:34.390705 30951 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:42763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786143361779964-25976-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786143361779964-25976-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.1.keytab
--principal=kudu/127.25.94.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786143361779964-25976-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.94.1
--webserver_port=0
--tserver_master_addrs=127.25.94.62:37089
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.1
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786143361779964-25976-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 22:56:34.392086 30951 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:56:34.393569 30951 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:56:34.408429 30959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:56:34.408524 30963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:56:34.408564 30960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:56:34.409327 30951 server_base.cc:1034] running on GCE node
I20260807 22:56:34.410004 30951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:56:34.410988 30951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:56:34.412223 30951 hybrid_clock.cc:648] HybridClock initialized: now 1786143394412176 us; error 61 us; skew 500 ppm
I20260807 22:56:34.412592 30951 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 22:56:34 dist-test-slave-z8x9 krb5kdc[30775](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143394, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:56:34.415993 30951 init.cc:377] Logged in from keytab as kudu/127.25.94.1@KRBTEST.COM (short username kudu)
I20260807 22:56:34.417663 30951 webserver.cc:460] Webserver started at http://127.25.94.1:37551/ using document root <none> and password file <none>
I20260807 22:56:34.418507 30951 fs_manager.cc:359] Metadata directory not provided
I20260807 22:56:34.418643 30951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:56:34.418959 30951 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:56:34.422057 30951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786143361779964-25976-0/minicluster-data/ts-0/data/instance:
uuid: "3bea1ae65dfb462ca8b0f10b3b2fd186"
format_stamp: "Formatted at 2026-08-07 22:56:34 on dist-test-slave-z8x9"
server_key: "96bbd5d57b110b326f805b7abdc8a43e"
server_key_iv: "387373feda3788414c03bd9144a2db47"
server_key_version: "encryptionkey@0"
I20260807 22:56:34.422976 30951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786143361779964-25976-0/minicluster-data/ts-0/wal/instance:
uuid: "3bea1ae65dfb462ca8b0f10b3b2fd186"
format_stamp: "Formatted at 2026-08-07 22:56:34 on dist-test-slave-z8x9"
server_key: "96bbd5d57b110b326f805b7abdc8a43e"
server_key_iv: "387373feda3788414c03bd9144a2db47"
server_key_version: "encryptionkey@0"
I20260807 22:56:34.428427 30951 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260807 22:56:34.432060 30971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:56:34.433688 30951 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 22:56:34.433878 30951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786143361779964-25976-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786143361779964-25976-0/minicluster-data/ts-0/wal
uuid: "3bea1ae65dfb462ca8b0f10b3b2fd186"
format_stamp: "Formatted at 2026-08-07 22:56:34 on dist-test-slave-z8x9"
server_key: "96bbd5d57b110b326f805b7abdc8a43e"
server_key_iv: "387373feda3788414c03bd9144a2db47"
server_key_version: "encryptionkey@0"
I20260807 22:56:34.434104 30951 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786143361779964-25976-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786143361779964-25976-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786143361779964-25976-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:56:34.446211 30951 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:56:34.451112 30951 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:56:34.451522 30951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:56:34.452625 30951 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:56:34.454072 30951 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:56:34.454149 30951 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:56:34.454303 30951 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:56:34.454372 30951 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:56:34.489132 30951 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.1:40555
I20260807 22:56:34.489225 31148 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.1:40555 every 8 connection(s)
I20260807 22:56:34.490567 30951 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786143361779964-25976-0/minicluster-data/ts-0/data/info.pb
I20260807 22:56:34.492136 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 30951
I20260807 22:56:34.492368 25976 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786143361779964-25976-0/minicluster-data/ts-0/wal/instance
I20260807 22:56:34.493315 25976 external_mini_cluster.cc:1428] Setting key bc91ffff513b211845aa715097e28e14
Aug 07 22:56:34 dist-test-slave-z8x9 krb5kdc[30775](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143394, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.1@KRBTEST.COM for kudu/127.25.94.62@KRBTEST.COM
I20260807 22:56:34.514238 31155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:34.493938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.1:59115 (local address 127.25.94.62:37089)
0807 22:56:34.494494 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:34.494500 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:34.498729 (+ 4229us) server_negotiation.cc:407] Connection header received
0807 22:56:34.498801 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:34.498808 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:34.498900 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:34.499018 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:56:34.500595 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:34.501757 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:34.503801 (+ 2044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:34.504208 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:34.504361 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:34.508114 (+ 3753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:56:34.508119 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:56:34.508123 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:56:34.508159 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:56:34.510902 (+ 2743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:34.511720 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:34.511730 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:34.511732 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:34.511863 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:34.512308 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:34.512314 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:34.512319 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:34.512636 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:56:34.512804 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:34.513605 (+ 801us) server_negotiation.cc:299] Negotiation successful
0807 22:56:34.513861 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":258,"threads_started":1}
WARNING: no policy specified for kudu/127.25.94.2@KRBTEST.COM; defaulting to no policy
I20260807 22:56:34.515270 31156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:34.494551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:37089 (local address 127.25.94.1:59115)
0807 22:56:34.495498 (+ 947us) negotiation.cc:107] Waiting for socket to connect
0807 22:56:34.495554 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0807 22:56:34.498354 (+ 2800us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:56:34.499270 (+ 916us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:56:34.499279 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:56:34.499820 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:56:34.500403 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:34.500420 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:34.501958 (+ 1538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:34.501966 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:34.503600 (+ 1634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:34.503619 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:34.504461 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:34.504471 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:34.504709 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:34.505475 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:56:34.505498 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:56:34.507858 (+ 2360us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:56:34.511065 (+ 3207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:56:34.511075 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:56:34.511094 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:56:34.511578 (+ 484us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:56:34.511979 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:56:34.511984 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:56:34.511988 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:56:34.512188 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:56:34.512781 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:56:34.512789 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:56:34.513145 (+ 356us) client_negotiation.cc:769] Sending connection context
0807 22:56:34.513506 (+ 361us) client_negotiation.cc:240] Negotiation successful
0807 22:56:34.514066 (+ 560us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":339,"threads_started":1}
I20260807 22:56:34.517124 31151 heartbeater.cc:347] Connected to a master server at 127.25.94.62:37089
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.2@KRBTEST.COM" created.
I20260807 22:56:34.517577 31151 heartbeater.cc:464] Registering TS with master...
I20260807 22:56:34.518585 31151 heartbeater.cc:511] Master 127.25.94.62:37089 requested a full tablet report, sending...
I20260807 22:56:34.521126 30850 ts_manager.cc:194] Registered new tserver with Master: 3bea1ae65dfb462ca8b0f10b3b2fd186 (127.25.94.1:40555)
I20260807 22:56:34.522863 30850 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.94.1@KRBTEST.COM'} at 127.25.94.1:59115
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.2.keytab.
Entry for principal kudu/127.25.94.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.2.keytab.
I20260807 22:56:34.544672 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786143361779964-25976-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786143361779964-25976-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786143361779964-25976-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786143361779964-25976-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.2:0
--local_ip_for_outbound_sockets=127.25.94.2
--webserver_interface=127.25.94.2
--webserver_port=0
--tserver_master_addrs=127.25.94.62:37089
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.2.keytab
--principal=kudu/127.25.94.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.20:42763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786143361779964-25976-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786143361779964-25976-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786143361779964-25976-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:56:34.895573 31160 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:56:34.895856 31160 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:56:34.896063 31160 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:56:34.903609 31160 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:56:34.903721 31160 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:56:34.903936 31160 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.2
I20260807 22:56:34.913635 31160 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:42763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786143361779964-25976-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786143361779964-25976-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.2.keytab
--principal=kudu/127.25.94.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786143361779964-25976-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.94.2
--webserver_port=0
--tserver_master_addrs=127.25.94.62:37089
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.2
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786143361779964-25976-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 22:56:34.914894 31160 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:56:34.916265 31160 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:56:34.930934 31168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:56:34.930866 31172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:56:34.930902 31169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:56:34.931975 31160 server_base.cc:1034] running on GCE node
I20260807 22:56:34.932642 31160 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:56:34.933771 31160 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:56:34.934975 31160 hybrid_clock.cc:648] HybridClock initialized: now 1786143394934929 us; error 30 us; skew 500 ppm
I20260807 22:56:34.935302 31160 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 22:56:34 dist-test-slave-z8x9 krb5kdc[30775](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143394, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:56:34.938977 31160 init.cc:377] Logged in from keytab as kudu/127.25.94.2@KRBTEST.COM (short username kudu)
I20260807 22:56:34.940356 31160 webserver.cc:460] Webserver started at http://127.25.94.2:42401/ using document root <none> and password file <none>
I20260807 22:56:34.941013 31160 fs_manager.cc:359] Metadata directory not provided
I20260807 22:56:34.941115 31160 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:56:34.941452 31160 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:56:34.944327 31160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786143361779964-25976-0/minicluster-data/ts-1/data/instance:
uuid: "91c57c41eda64b29813e6a416ad1d920"
format_stamp: "Formatted at 2026-08-07 22:56:34 on dist-test-slave-z8x9"
server_key: "b1b52cc7258cc92a617abb77778463dd"
server_key_iv: "d7ba5dad25ff9f066d40a275887c2815"
server_key_version: "encryptionkey@0"
I20260807 22:56:34.945075 31160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786143361779964-25976-0/minicluster-data/ts-1/wal/instance:
uuid: "91c57c41eda64b29813e6a416ad1d920"
format_stamp: "Formatted at 2026-08-07 22:56:34 on dist-test-slave-z8x9"
server_key: "b1b52cc7258cc92a617abb77778463dd"
server_key_iv: "d7ba5dad25ff9f066d40a275887c2815"
server_key_version: "encryptionkey@0"
I20260807 22:56:34.950858 31160 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260807 22:56:34.954665 31181 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:56:34.956018 31160 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 22:56:34.956166 31160 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786143361779964-25976-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786143361779964-25976-0/minicluster-data/ts-1/wal
uuid: "91c57c41eda64b29813e6a416ad1d920"
format_stamp: "Formatted at 2026-08-07 22:56:34 on dist-test-slave-z8x9"
server_key: "b1b52cc7258cc92a617abb77778463dd"
server_key_iv: "d7ba5dad25ff9f066d40a275887c2815"
server_key_version: "encryptionkey@0"
I20260807 22:56:34.956357 31160 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786143361779964-25976-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786143361779964-25976-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786143361779964-25976-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:56:34.981755 31160 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:56:34.986583 31160 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:56:34.986938 31160 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:56:34.987916 31160 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:56:34.989326 31160 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:56:34.989405 31160 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:56:34.989492 31160 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:56:34.989537 31160 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:56:35.025465 31160 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.2:44327
I20260807 22:56:35.025516 31342 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.2:44327 every 8 connection(s)
I20260807 22:56:35.026826 31160 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786143361779964-25976-0/minicluster-data/ts-1/data/info.pb
I20260807 22:56:35.027841 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 31160
I20260807 22:56:35.028065 25976 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786143361779964-25976-0/minicluster-data/ts-1/wal/instance
I20260807 22:56:35.028687 25976 external_mini_cluster.cc:1428] Setting key 9b9f06ed0fa6e3004b50915d5dae49f7
Aug 07 22:56:35 dist-test-slave-z8x9 krb5kdc[30775](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143394, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.2@KRBTEST.COM for kudu/127.25.94.62@KRBTEST.COM
I20260807 22:56:35.050653 31345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:35.029989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.2:55113 (local address 127.25.94.62:37089)
0807 22:56:35.030439 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:35.030445 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:35.031465 (+ 1020us) server_negotiation.cc:407] Connection header received
0807 22:56:35.033387 (+ 1922us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:35.033393 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:35.033487 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:35.033611 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:56:35.035205 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:35.035939 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:35.037901 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:35.038394 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:35.038536 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:35.042786 (+ 4250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:56:35.042796 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:56:35.042804 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:56:35.042854 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:56:35.046805 (+ 3951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:35.047729 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:35.047738 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:35.047744 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:35.047926 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:35.048495 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:35.048503 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:35.048508 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:35.048933 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:56:35.049127 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:35.049937 (+ 810us) server_negotiation.cc:299] Negotiation successful
0807 22:56:35.050203 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":226,"threads_started":1}
I20260807 22:56:35.051378 31346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:35.030443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:37089 (local address 127.25.94.2:55113)
0807 22:56:35.031316 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0807 22:56:35.031355 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0807 22:56:35.033041 (+ 1686us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:56:35.033864 (+ 823us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:56:35.033880 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:56:35.034418 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:56:35.035020 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:35.035039 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:35.036106 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:35.036116 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:35.037746 (+ 1630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:35.037764 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:35.038637 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:35.038645 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:35.038920 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:35.039597 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:56:35.039632 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:56:35.042511 (+ 2879us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:56:35.047024 (+ 4513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:56:35.047032 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:56:35.047048 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:56:35.047561 (+ 513us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:56:35.048106 (+ 545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:56:35.048111 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:56:35.048114 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:56:35.048345 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:56:35.049081 (+ 736us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:56:35.049088 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:56:35.049508 (+ 420us) client_negotiation.cc:769] Sending connection context
0807 22:56:35.049756 (+ 248us) client_negotiation.cc:240] Negotiation successful
0807 22:56:35.050177 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":649,"thread_start_us":384,"threads_started":1}
WARNING: no policy specified for kudu/127.25.94.3@KRBTEST.COM; defaulting to no policy
I20260807 22:56:35.053040 31343 heartbeater.cc:347] Connected to a master server at 127.25.94.62:37089
I20260807 22:56:35.053419 31343 heartbeater.cc:464] Registering TS with master...
I20260807 22:56:35.054185 31343 heartbeater.cc:511] Master 127.25.94.62:37089 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.3@KRBTEST.COM" created.
I20260807 22:56:35.055828 30850 ts_manager.cc:194] Registered new tserver with Master: 91c57c41eda64b29813e6a416ad1d920 (127.25.94.2:44327)
I20260807 22:56:35.057025 30850 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.94.2@KRBTEST.COM'} at 127.25.94.2:55113
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.3.keytab.
Entry for principal kudu/127.25.94.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.3.keytab.
I20260807 22:56:35.085006 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786143361779964-25976-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786143361779964-25976-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786143361779964-25976-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786143361779964-25976-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.3:0
--local_ip_for_outbound_sockets=127.25.94.3
--webserver_interface=127.25.94.3
--webserver_port=0
--tserver_master_addrs=127.25.94.62:37089
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.3.keytab
--principal=kudu/127.25.94.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.20:42763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786143361779964-25976-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786143361779964-25976-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786143361779964-25976-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:56:35.435011 31351 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:56:35.435313 31351 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:56:35.435475 31351 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:56:35.443120 31351 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:56:35.443241 31351 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:56:35.443465 31351 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.3
I20260807 22:56:35.453423 31351 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:42763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786143361779964-25976-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786143361779964-25976-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.3.keytab
--principal=kudu/127.25.94.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786143361779964-25976-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.94.3
--webserver_port=0
--tserver_master_addrs=127.25.94.62:37089
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.3
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786143361779964-25976-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 22:56:35.454824 31351 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:56:35.456218 31351 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:56:35.469933 31359 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:56:35.469875 31363 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:56:35.469898 31371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:56:35.470851 31351 server_base.cc:1034] running on GCE node
I20260807 22:56:35.471431 31351 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:56:35.472390 31351 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:56:35.473618 31351 hybrid_clock.cc:648] HybridClock initialized: now 1786143395473563 us; error 41 us; skew 500 ppm
I20260807 22:56:35.473984 31351 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 22:56:35 dist-test-slave-z8x9 krb5kdc[30775](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143395, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:56:35.477566 31351 init.cc:377] Logged in from keytab as kudu/127.25.94.3@KRBTEST.COM (short username kudu)
I20260807 22:56:35.479140 31351 webserver.cc:460] Webserver started at http://127.25.94.3:39333/ using document root <none> and password file <none>
I20260807 22:56:35.479871 31351 fs_manager.cc:359] Metadata directory not provided
I20260807 22:56:35.479981 31351 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:56:35.480247 31351 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:56:35.483222 31351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786143361779964-25976-0/minicluster-data/ts-2/data/instance:
uuid: "bafe2a12e7b74c7aa6def454a0587680"
format_stamp: "Formatted at 2026-08-07 22:56:35 on dist-test-slave-z8x9"
server_key: "8f1ef17bbdf575338c636e9b2ee65fc3"
server_key_iv: "58cc1311c9d2543e476aa79eeac0404b"
server_key_version: "encryptionkey@0"
I20260807 22:56:35.483958 31351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786143361779964-25976-0/minicluster-data/ts-2/wal/instance:
uuid: "bafe2a12e7b74c7aa6def454a0587680"
format_stamp: "Formatted at 2026-08-07 22:56:35 on dist-test-slave-z8x9"
server_key: "8f1ef17bbdf575338c636e9b2ee65fc3"
server_key_iv: "58cc1311c9d2543e476aa79eeac0404b"
server_key_version: "encryptionkey@0"
I20260807 22:56:35.489706 31351 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260807 22:56:35.494357 31379 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:56:35.495796 31351 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 22:56:35.495939 31351 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786143361779964-25976-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786143361779964-25976-0/minicluster-data/ts-2/wal
uuid: "bafe2a12e7b74c7aa6def454a0587680"
format_stamp: "Formatted at 2026-08-07 22:56:35 on dist-test-slave-z8x9"
server_key: "8f1ef17bbdf575338c636e9b2ee65fc3"
server_key_iv: "58cc1311c9d2543e476aa79eeac0404b"
server_key_version: "encryptionkey@0"
I20260807 22:56:35.496106 31351 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786143361779964-25976-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786143361779964-25976-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786143361779964-25976-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:56:35.510193 31351 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:56:35.514634 31351 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:56:35.514894 31351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:56:35.515794 31351 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:56:35.517218 31351 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:56:35.517306 31351 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260807 22:56:35.517402 31351 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:56:35.517448 31351 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:56:35.526052 31151 heartbeater.cc:503] Master 127.25.94.62:37089 was elected leader, sending a full tablet report...
I20260807 22:56:35.551529 31351 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.3:39781
I20260807 22:56:35.551553 31549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.3:39781 every 8 connection(s)
I20260807 22:56:35.553711 31351 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786143361779964-25976-0/minicluster-data/ts-2/data/info.pb
I20260807 22:56:35.559831 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 31351
I20260807 22:56:35.560047 25976 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786143361779964-25976-0/minicluster-data/ts-2/wal/instance
I20260807 22:56:35.560650 25976 external_mini_cluster.cc:1428] Setting key a534db5197df5f19a64944b104cc75e9
Aug 07 22:56:35 dist-test-slave-z8x9 krb5kdc[30775](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143395, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.3@KRBTEST.COM for kudu/127.25.94.62@KRBTEST.COM
I20260807 22:56:35.577559 31557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:35.556605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.3:33127 (local address 127.25.94.62:37089)
0807 22:56:35.557025 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:35.557030 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:35.558115 (+ 1085us) server_negotiation.cc:407] Connection header received
0807 22:56:35.560263 (+ 2148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:35.560268 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:35.560352 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:35.560506 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:56:35.564887 (+ 4381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:35.565829 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:35.567400 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:35.567783 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:35.567885 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:35.571465 (+ 3580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:56:35.571470 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:56:35.571474 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:56:35.571504 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:56:35.574033 (+ 2529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:35.574817 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:35.574822 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:35.574826 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:35.575003 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:35.575614 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:35.575619 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:35.575621 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:35.576028 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:56:35.576278 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:35.576945 (+ 667us) server_negotiation.cc:299] Negotiation successful
0807 22:56:35.577127 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":205,"threads_started":1}
I20260807 22:56:35.578564 31558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:35.557233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:37089 (local address 127.25.94.3:33127)
0807 22:56:35.557954 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0807 22:56:35.558003 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0807 22:56:35.559934 (+ 1931us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:56:35.560817 (+ 883us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:56:35.560827 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:56:35.564008 (+ 3181us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:56:35.564684 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:35.564705 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:35.565977 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:35.565981 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:35.567254 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:35.567273 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:35.567984 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:35.567988 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:35.568179 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:35.568921 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:56:35.568944 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:56:35.571255 (+ 2311us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:56:35.574164 (+ 2909us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:56:35.574173 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:56:35.574191 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:56:35.574660 (+ 469us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:56:35.575162 (+ 502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:56:35.575166 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:56:35.575169 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:56:35.575449 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:56:35.576193 (+ 744us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:56:35.576200 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:56:35.576589 (+ 389us) client_negotiation.cc:769] Sending connection context
0807 22:56:35.576823 (+ 234us) client_negotiation.cc:240] Negotiation successful
0807 22:56:35.577458 (+ 635us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":284,"threads_started":1}
I20260807 22:56:35.580056 31550 heartbeater.cc:347] Connected to a master server at 127.25.94.62:37089
I20260807 22:56:35.580399 31550 heartbeater.cc:464] Registering TS with master...
I20260807 22:56:35.581230 31550 heartbeater.cc:511] Master 127.25.94.62:37089 requested a full tablet report, sending...
I20260807 22:56:35.582669 30850 ts_manager.cc:194] Registered new tserver with Master: bafe2a12e7b74c7aa6def454a0587680 (127.25.94.3:39781)
I20260807 22:56:35.583483 30850 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.94.3@KRBTEST.COM'} at 127.25.94.3:33127
I20260807 22:56:35.588886 25976 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 07 22:56:35 dist-test-slave-z8x9 krb5kdc[30775](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143395, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 07 22:56:35 dist-test-slave-z8x9 krb5kdc[30775](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143395, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.25.94.62@KRBTEST.COM
I20260807 22:56:35.642180 31557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:35.627764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60890 (local address 127.25.94.62:37089)
0807 22:56:35.627949 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:35.627954 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:35.628477 (+ 523us) server_negotiation.cc:407] Connection header received
0807 22:56:35.628636 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:35.628641 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:35.628708 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:35.628803 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:56:35.630288 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:35.631028 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:35.632258 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:35.632644 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:35.632738 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:35.636761 (+ 4023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:56:35.636767 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:56:35.636771 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:56:35.636806 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:56:35.639702 (+ 2896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:35.640346 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:35.640350 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:35.640352 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:35.640489 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:35.640847 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:35.640850 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:35.640852 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:35.641277 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:56:35.641487 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:35.641742 (+ 255us) server_negotiation.cc:299] Negotiation successful
0807 22:56:35.641915 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260807 22:56:35.642946 30850 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:60890
I20260807 22:56:35.644891 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 30951
Aug 07 22:56:35 dist-test-slave-z8x9 krb5kdc[30775](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143395, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.25.94.1@KRBTEST.COM
I20260807 22:56:35.661377 31571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:35.645925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36100 (local address 127.25.94.1:40555)
0807 22:56:35.646534 (+ 609us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:35.646542 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:35.646566 (+ 24us) server_negotiation.cc:407] Connection header received
0807 22:56:35.646628 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:35.646634 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:35.646842 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:35.646971 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:56:35.648101 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:35.649255 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:35.650452 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:35.651032 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:35.651131 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:35.655277 (+ 4146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:56:35.655290 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:56:35.655310 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:56:35.655346 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:56:35.658453 (+ 3107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:35.659144 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:35.659160 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:35.659162 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:35.659309 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:35.659779 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:35.659784 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:35.659787 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:35.660156 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:56:35.660436 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:35.660659 (+ 223us) server_negotiation.cc:299] Negotiation successful
0807 22:56:35.660914 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":462,"thread_start_us":275,"threads_started":1}
W20260807 22:56:35.662871 31144 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:36100
W20260807 22:56:35.663965 25976 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 30951: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 22:56:35.664192 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 30951
I20260807 22:56:35.689471 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 31160
Aug 07 22:56:35 dist-test-slave-z8x9 krb5kdc[30775](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143395, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.25.94.2@KRBTEST.COM
I20260807 22:56:35.705857 31572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:35.690379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37240 (local address 127.25.94.2:44327)
0807 22:56:35.690992 (+ 613us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:35.691000 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:35.691027 (+ 27us) server_negotiation.cc:407] Connection header received
0807 22:56:35.691089 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:35.691095 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:35.691312 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:35.691461 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:56:35.692723 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:35.693782 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:35.695044 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:35.695593 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:35.695688 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:35.699940 (+ 4252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:56:35.699952 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:56:35.699966 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:56:35.700013 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:56:35.703224 (+ 3211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:35.703845 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:35.703852 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:35.703854 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:35.703985 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:35.704291 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:35.704295 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:35.704296 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:35.704648 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:56:35.704881 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:35.705163 (+ 282us) server_negotiation.cc:299] Negotiation successful
0807 22:56:35.705461 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":439,"thread_start_us":324,"threads_started":1}
W20260807 22:56:35.707115 31337 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:37240
W20260807 22:56:35.708211 25976 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 31160: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 22:56:35.708384 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 31160
I20260807 22:56:35.733878 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 31351
Aug 07 22:56:35 dist-test-slave-z8x9 krb5kdc[30775](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143395, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.25.94.3@KRBTEST.COM
I20260807 22:56:35.750988 31573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:35.734853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37064 (local address 127.25.94.3:39781)
0807 22:56:35.735538 (+ 685us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:35.735545 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:35.735573 (+ 28us) server_negotiation.cc:407] Connection header received
0807 22:56:35.735636 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:35.735642 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:35.735863 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:35.735997 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:56:35.737598 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:35.738776 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:35.740009 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:35.740561 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:35.740665 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:35.745788 (+ 5123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:56:35.745801 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:56:35.745818 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:56:35.745853 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:56:35.748495 (+ 2642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:35.749135 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:35.749142 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:35.749144 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:35.749319 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:35.749647 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:35.749651 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:35.749652 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:35.749983 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:56:35.750209 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:35.750453 (+ 244us) server_negotiation.cc:299] Negotiation successful
0807 22:56:35.750680 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":362,"threads_started":1}
W20260807 22:56:35.752300 31545 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:37064
W20260807 22:56:35.753350 25976 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 31351: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 22:56:35.753542 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 31351
I20260807 22:56:35.774304 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 30796
I20260807 22:56:35.775487 30895 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 22:56:35.878762 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 30796
2026-08-07T22:56:35Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (4025 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786143361779964-25976-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 22:56:35 dist-test-slave-z8x9 krb5kdc[31578](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 22:56:35 dist-test-slave-z8x9 krb5kdc[31578](info): set up 2 sockets
Aug 07 22:56:35 dist-test-slave-z8x9 krb5kdc[31578](info): commencing operation
krb5kdc: starting...
W20260807 22:56:37.240549 25976 mini_kdc.cc:121] Time spent starting KDC: real 1.312s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786143361779964-25976-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786143361779964-25976-0/krb5kdc/test-user.keytab.
Aug 07 22:56:37 dist-test-slave-z8x9 krb5kdc[31578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143397, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T22:56:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T22:56:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.94.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.62.keytab.
Entry for principal kudu/127.25.94.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.62.keytab.
I20260807 22:56:37.473493 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786143361779964-25976-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786143361779964-25976-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786143361779964-25976-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.62:36195
--webserver_interface=127.25.94.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.62.keytab
--principal=kudu/127.25.94.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.20:32895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.94.62:36195
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786143361779964-25976-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786143361779964-25976-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786143361779964-25976-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:56:37.815043 31597 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:56:37.815400 31597 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:56:37.815565 31597 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:56:37.822844 31597 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 22:56:37.823005 31597 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:56:37.823122 31597 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:56:37.823230 31597 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 22:56:37.823318 31597 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 22:56:37.833614 31597 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:32895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786143361779964-25976-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786143361779964-25976-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.94.62:36195
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.62.keytab
--principal=kudu/127.25.94.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.62:36195
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.94.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786143361779964-25976-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 22:56:37.835132 31597 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:56:37.836489 31597 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:56:37.850015 31609 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:56:37.849987 31614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:56:37.849949 31610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:56:37.850855 31597 server_base.cc:1034] running on GCE node
I20260807 22:56:37.851768 31597 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:56:37.853389 31597 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:56:37.854604 31597 hybrid_clock.cc:648] HybridClock initialized: now 1786143397854556 us; error 36 us; skew 500 ppm
I20260807 22:56:37.855175 31597 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 22:56:37 dist-test-slave-z8x9 krb5kdc[31578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143397, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:56:37.861994 31597 init.cc:377] Logged in from keytab as kudu/127.25.94.62@KRBTEST.COM (short username kudu)
I20260807 22:56:37.863726 31597 webserver.cc:460] Webserver started at http://127.25.94.62:42441/ using document root <none> and password file <none>
I20260807 22:56:37.864488 31597 fs_manager.cc:359] Metadata directory not provided
I20260807 22:56:37.864632 31597 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:56:37.864993 31597 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:56:37.868206 31597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786143361779964-25976-0/minicluster-data/master-0/data/instance:
uuid: "1d8b02334123407caf2e7b1edb65a1b3"
format_stamp: "Formatted at 2026-08-07 22:56:37 on dist-test-slave-z8x9"
server_key: "aa34d78404ab2abc8fc0cca83cc2e140"
server_key_iv: "8edb3063d6c912f64bc0f64dc55c86fa"
server_key_version: "encryptionkey@0"
I20260807 22:56:37.869089 31597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786143361779964-25976-0/minicluster-data/master-0/wal/instance:
uuid: "1d8b02334123407caf2e7b1edb65a1b3"
format_stamp: "Formatted at 2026-08-07 22:56:37 on dist-test-slave-z8x9"
server_key: "aa34d78404ab2abc8fc0cca83cc2e140"
server_key_iv: "8edb3063d6c912f64bc0f64dc55c86fa"
server_key_version: "encryptionkey@0"
I20260807 22:56:37.874615 31597 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260807 22:56:37.878289 31624 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:56:37.879657 31597 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260807 22:56:37.879851 31597 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786143361779964-25976-0/minicluster-data/master-0/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786143361779964-25976-0/minicluster-data/master-0/wal
uuid: "1d8b02334123407caf2e7b1edb65a1b3"
format_stamp: "Formatted at 2026-08-07 22:56:37 on dist-test-slave-z8x9"
server_key: "aa34d78404ab2abc8fc0cca83cc2e140"
server_key_iv: "8edb3063d6c912f64bc0f64dc55c86fa"
server_key_version: "encryptionkey@0"
I20260807 22:56:37.880062 31597 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786143361779964-25976-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786143361779964-25976-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786143361779964-25976-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:56:37.900005 31597 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:56:37.904268 31597 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:56:37.904604 31597 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:56:37.923861 31597 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.62:36195
I20260807 22:56:37.923909 31696 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.62:36195 every 8 connection(s)
I20260807 22:56:37.925351 31597 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
I20260807 22:56:37.926537 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 31597
I20260807 22:56:37.926790 25976 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786143361779964-25976-0/minicluster-data/master-0/wal/instance
I20260807 22:56:37.927451 25976 external_mini_cluster.cc:1428] Setting key 801efdae2e810096a5eae68216e8cb6a
I20260807 22:56:37.929973 31697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 07 22:56:37 dist-test-slave-z8x9 krb5kdc[31578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143397, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.62@KRBTEST.COM
I20260807 22:56:37.943032 31697 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d8b02334123407caf2e7b1edb65a1b3: Bootstrap starting.
I20260807 22:56:37.946477 31697 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1d8b02334123407caf2e7b1edb65a1b3: Neither blocks nor log segments found. Creating new log.
I20260807 22:56:37.947570 31697 log.cc:824] T 00000000000000000000000000000000 P 1d8b02334123407caf2e7b1edb65a1b3: Log is configured to *not* fsync() on all Append() calls
I20260807 22:56:37.951156 31697 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d8b02334123407caf2e7b1edb65a1b3: No bootstrap required, opened a new log
I20260807 22:56:37.952096 31703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:37.929548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51996 (local address 127.25.94.62:36195)
0807 22:56:37.930458 (+ 910us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:37.930474 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:37.930523 (+ 49us) server_negotiation.cc:407] Connection header received
0807 22:56:37.932850 (+ 2327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:37.932877 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:37.933425 (+ 548us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:37.933848 (+ 423us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:56:37.935288 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:37.937484 (+ 2196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:37.938699 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:37.939739 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:37.939938 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:37.944256 (+ 4318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:56:37.944264 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:56:37.944278 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:56:37.944315 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:56:37.947549 (+ 3234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:37.948465 (+ 916us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:37.948481 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:37.948491 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:37.948715 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:37.949267 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:37.949271 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:37.949273 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:37.949792 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:56:37.950012 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:37.950553 (+ 541us) server_negotiation.cc:299] Negotiation successful
0807 22:56:37.950939 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":617,"thread_start_us":379,"threads_started":1}
I20260807 22:56:37.959303 31697 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1d8b02334123407caf2e7b1edb65a1b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d8b02334123407caf2e7b1edb65a1b3" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 36195 } }
I20260807 22:56:37.959748 31697 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1d8b02334123407caf2e7b1edb65a1b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:56:37.959879 31697 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1d8b02334123407caf2e7b1edb65a1b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d8b02334123407caf2e7b1edb65a1b3, State: Initialized, Role: FOLLOWER
I20260807 22:56:37.960453 31697 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d8b02334123407caf2e7b1edb65a1b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d8b02334123407caf2e7b1edb65a1b3" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 36195 } }
I20260807 22:56:37.960739 31697 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1d8b02334123407caf2e7b1edb65a1b3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 22:56:37.960904 31697 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1d8b02334123407caf2e7b1edb65a1b3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 22:56:37.961112 31697 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1d8b02334123407caf2e7b1edb65a1b3 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:56:37.963497 31697 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1d8b02334123407caf2e7b1edb65a1b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d8b02334123407caf2e7b1edb65a1b3" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 36195 } }
I20260807 22:56:37.964068 31697 leader_election.cc:302] T 00000000000000000000000000000000 P 1d8b02334123407caf2e7b1edb65a1b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1d8b02334123407caf2e7b1edb65a1b3; no voters:
I20260807 22:56:37.964681 31697 leader_election.cc:288] T 00000000000000000000000000000000 P 1d8b02334123407caf2e7b1edb65a1b3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 22:56:37.964838 31705 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1d8b02334123407caf2e7b1edb65a1b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:56:37.965608 31705 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1d8b02334123407caf2e7b1edb65a1b3 [term 1 LEADER]: Becoming Leader. State: Replica: 1d8b02334123407caf2e7b1edb65a1b3, State: Running, Role: LEADER
I20260807 22:56:37.966207 31705 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1d8b02334123407caf2e7b1edb65a1b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d8b02334123407caf2e7b1edb65a1b3" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 36195 } }
I20260807 22:56:37.966982 31697 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1d8b02334123407caf2e7b1edb65a1b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 22:56:37.969461 31707 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d8b02334123407caf2e7b1edb65a1b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d8b02334123407caf2e7b1edb65a1b3. Latest consensus state: current_term: 1 leader_uuid: "1d8b02334123407caf2e7b1edb65a1b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d8b02334123407caf2e7b1edb65a1b3" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 36195 } } }
I20260807 22:56:37.969878 31707 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d8b02334123407caf2e7b1edb65a1b3 [sys.catalog]: This master's current role is: LEADER
I20260807 22:56:37.969873 31706 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d8b02334123407caf2e7b1edb65a1b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1d8b02334123407caf2e7b1edb65a1b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d8b02334123407caf2e7b1edb65a1b3" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 36195 } } }
I20260807 22:56:37.970341 31706 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d8b02334123407caf2e7b1edb65a1b3 [sys.catalog]: This master's current role is: LEADER
I20260807 22:56:37.971909 31716 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 22:56:37.977200 31716 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 22:56:37.987040 31716 catalog_manager.cc:1347] Generated new cluster ID: 16d1d25ca36e48d880af9b2d147429bc
I20260807 22:56:37.987198 31716 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 22:56:38.000720 31716 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 22:56:38.002661 31716 catalog_manager.cc:1504] Loading token signing keys...
I20260807 22:56:38.009034 31716 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1d8b02334123407caf2e7b1edb65a1b3: Generated new TSK 0
I20260807 22:56:38.009914 31716 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.94.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.1.keytab.
Entry for principal kudu/127.25.94.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.1.keytab.
I20260807 22:56:38.077427 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786143361779964-25976-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786143361779964-25976-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786143361779964-25976-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786143361779964-25976-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.1:0
--local_ip_for_outbound_sockets=127.25.94.1
--webserver_interface=127.25.94.1
--webserver_port=0
--tserver_master_addrs=127.25.94.62:36195
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.1.keytab
--principal=kudu/127.25.94.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.20:32895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786143361779964-25976-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786143361779964-25976-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786143361779964-25976-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:56:38.438886 31735 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:56:38.439260 31735 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:56:38.439440 31735 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:56:38.447024 31735 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:56:38.447197 31735 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:56:38.447377 31735 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.1
I20260807 22:56:38.457965 31735 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:32895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786143361779964-25976-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786143361779964-25976-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.1.keytab
--principal=kudu/127.25.94.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786143361779964-25976-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.94.1
--webserver_port=0
--tserver_master_addrs=127.25.94.62:36195
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.1
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786143361779964-25976-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 22:56:38.459399 31735 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:56:38.460748 31735 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:56:38.474732 31743 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:56:38.474789 31744 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:56:38.474812 31746 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:56:38.475593 31735 server_base.cc:1034] running on GCE node
I20260807 22:56:38.476177 31735 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:56:38.477237 31735 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:56:38.478472 31735 hybrid_clock.cc:648] HybridClock initialized: now 1786143398478417 us; error 46 us; skew 500 ppm
I20260807 22:56:38.478826 31735 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 22:56:38 dist-test-slave-z8x9 krb5kdc[31578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143398, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:56:38.482304 31735 init.cc:377] Logged in from keytab as kudu/127.25.94.1@KRBTEST.COM (short username kudu)
I20260807 22:56:38.483912 31735 webserver.cc:460] Webserver started at http://127.25.94.1:41781/ using document root <none> and password file <none>
I20260807 22:56:38.484675 31735 fs_manager.cc:359] Metadata directory not provided
I20260807 22:56:38.484834 31735 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:56:38.485260 31735 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:56:38.488268 31735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786143361779964-25976-0/minicluster-data/ts-0/data/instance:
uuid: "18c1d3b99b5b4ecb95d40ffdb1131057"
format_stamp: "Formatted at 2026-08-07 22:56:38 on dist-test-slave-z8x9"
server_key: "24d6e41e546a1d23ba309057624b6f29"
server_key_iv: "6fbcfef7731e16fee79d19590ecad637"
server_key_version: "encryptionkey@0"
I20260807 22:56:38.489099 31735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786143361779964-25976-0/minicluster-data/ts-0/wal/instance:
uuid: "18c1d3b99b5b4ecb95d40ffdb1131057"
format_stamp: "Formatted at 2026-08-07 22:56:38 on dist-test-slave-z8x9"
server_key: "24d6e41e546a1d23ba309057624b6f29"
server_key_iv: "6fbcfef7731e16fee79d19590ecad637"
server_key_version: "encryptionkey@0"
I20260807 22:56:38.494448 31735 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260807 22:56:38.497933 31758 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:56:38.499341 31735 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 22:56:38.499529 31735 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786143361779964-25976-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786143361779964-25976-0/minicluster-data/ts-0/wal
uuid: "18c1d3b99b5b4ecb95d40ffdb1131057"
format_stamp: "Formatted at 2026-08-07 22:56:38 on dist-test-slave-z8x9"
server_key: "24d6e41e546a1d23ba309057624b6f29"
server_key_iv: "6fbcfef7731e16fee79d19590ecad637"
server_key_version: "encryptionkey@0"
I20260807 22:56:38.499758 31735 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786143361779964-25976-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786143361779964-25976-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786143361779964-25976-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:56:38.515475 31735 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:56:38.519618 31735 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:56:38.519922 31735 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:56:38.520893 31735 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:56:38.522313 31735 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:56:38.522390 31735 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:56:38.522519 31735 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:56:38.522588 31735 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:56:38.557435 31735 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.1:35043
I20260807 22:56:38.557575 31944 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.1:35043 every 8 connection(s)
I20260807 22:56:38.558801 31735 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786143361779964-25976-0/minicluster-data/ts-0/data/info.pb
I20260807 22:56:38.562278 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 31735
I20260807 22:56:38.562569 25976 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786143361779964-25976-0/minicluster-data/ts-0/wal/instance
I20260807 22:56:38.563434 25976 external_mini_cluster.cc:1428] Setting key 0efcce347e403709901aba7d48614503
Aug 07 22:56:38 dist-test-slave-z8x9 krb5kdc[31578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143398, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.1@KRBTEST.COM for kudu/127.25.94.62@KRBTEST.COM
I20260807 22:56:38.580732 31951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:38.561438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.1:55453 (local address 127.25.94.62:36195)
0807 22:56:38.561947 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:38.561952 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:38.563158 (+ 1206us) server_negotiation.cc:407] Connection header received
0807 22:56:38.565770 (+ 2612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:38.565776 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:38.565862 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:38.565970 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:56:38.568020 (+ 2050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:38.568819 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:38.570635 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:38.571037 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:38.571198 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:38.574807 (+ 3609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:56:38.574812 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:56:38.574815 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:56:38.574847 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:56:38.577581 (+ 2734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:38.578353 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:38.578358 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:38.578360 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:38.578480 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:38.578939 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:38.578948 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:38.578950 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:38.579226 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:56:38.579385 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:38.580114 (+ 729us) server_negotiation.cc:299] Negotiation successful
0807 22:56:38.580349 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":270,"threads_started":1}
I20260807 22:56:38.581840 31952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:38.562024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:36195 (local address 127.25.94.1:55453)
0807 22:56:38.562979 (+ 955us) negotiation.cc:107] Waiting for socket to connect
0807 22:56:38.563019 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0807 22:56:38.565406 (+ 2387us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:56:38.566323 (+ 917us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:56:38.566333 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:56:38.567115 (+ 782us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:56:38.567843 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:38.567862 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:38.569030 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:38.569038 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:38.570470 (+ 1432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:38.570483 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:38.571287 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:38.571297 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:38.571547 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:38.572283 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:56:38.572305 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:56:38.574568 (+ 2263us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:56:38.577744 (+ 3176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:56:38.577753 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:56:38.577768 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:56:38.578211 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:56:38.578596 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:56:38.578601 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:56:38.578605 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:56:38.578811 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:56:38.579361 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:56:38.579369 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:56:38.579726 (+ 357us) client_negotiation.cc:769] Sending connection context
0807 22:56:38.579995 (+ 269us) client_negotiation.cc:240] Negotiation successful
0807 22:56:38.580558 (+ 563us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":716,"thread_start_us":421,"threads_started":1}
I20260807 22:56:38.583731 31948 heartbeater.cc:347] Connected to a master server at 127.25.94.62:36195
I20260807 22:56:38.584116 31948 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.25.94.2@KRBTEST.COM; defaulting to no policy
I20260807 22:56:38.585035 31948 heartbeater.cc:511] Master 127.25.94.62:36195 requested a full tablet report, sending...
I20260807 22:56:38.587397 31644 ts_manager.cc:194] Registered new tserver with Master: 18c1d3b99b5b4ecb95d40ffdb1131057 (127.25.94.1:35043)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.2@KRBTEST.COM" created.
I20260807 22:56:38.588960 31644 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.94.1@KRBTEST.COM'} at 127.25.94.1:55453
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.2.keytab.
Entry for principal kudu/127.25.94.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.2.keytab.
I20260807 22:56:38.616183 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786143361779964-25976-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786143361779964-25976-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786143361779964-25976-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786143361779964-25976-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.2:0
--local_ip_for_outbound_sockets=127.25.94.2
--webserver_interface=127.25.94.2
--webserver_port=0
--tserver_master_addrs=127.25.94.62:36195
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.2.keytab
--principal=kudu/127.25.94.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.20:32895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786143361779964-25976-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786143361779964-25976-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786143361779964-25976-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:56:38.960791 31955 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:56:38.961089 31955 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:56:38.961244 31955 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:56:38.968832 31955 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:56:38.968963 31955 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:56:38.969079 31955 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.2
I20260807 22:56:38.979478 31955 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:32895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786143361779964-25976-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786143361779964-25976-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.2.keytab
--principal=kudu/127.25.94.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786143361779964-25976-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.94.2
--webserver_port=0
--tserver_master_addrs=127.25.94.62:36195
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.2
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786143361779964-25976-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 22:56:38.980902 31955 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:56:38.982239 31955 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:56:38.996272 31964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:56:38.996228 31960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:56:38.996325 31961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:56:38.997097 31955 server_base.cc:1034] running on GCE node
I20260807 22:56:38.997784 31955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:56:38.998746 31955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:56:38.999981 31955 hybrid_clock.cc:648] HybridClock initialized: now 1786143398999925 us; error 47 us; skew 500 ppm
I20260807 22:56:39.000327 31955 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 22:56:38 dist-test-slave-z8x9 krb5kdc[31578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143398, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:56:39.003705 31955 init.cc:377] Logged in from keytab as kudu/127.25.94.2@KRBTEST.COM (short username kudu)
I20260807 22:56:39.005254 31955 webserver.cc:460] Webserver started at http://127.25.94.2:41271/ using document root <none> and password file <none>
I20260807 22:56:39.005916 31955 fs_manager.cc:359] Metadata directory not provided
I20260807 22:56:39.006021 31955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:56:39.006283 31955 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:56:39.009203 31955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786143361779964-25976-0/minicluster-data/ts-1/data/instance:
uuid: "407469795bc444cd9ff14296135e0ded"
format_stamp: "Formatted at 2026-08-07 22:56:39 on dist-test-slave-z8x9"
server_key: "120f42a5c29e8baa5b84745bfe806501"
server_key_iv: "7e396d9ea0a052f06258cc6edfeb161c"
server_key_version: "encryptionkey@0"
I20260807 22:56:39.009994 31955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786143361779964-25976-0/minicluster-data/ts-1/wal/instance:
uuid: "407469795bc444cd9ff14296135e0ded"
format_stamp: "Formatted at 2026-08-07 22:56:39 on dist-test-slave-z8x9"
server_key: "120f42a5c29e8baa5b84745bfe806501"
server_key_iv: "7e396d9ea0a052f06258cc6edfeb161c"
server_key_version: "encryptionkey@0"
I20260807 22:56:39.015115 31955 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260807 22:56:39.018599 31972 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:56:39.019882 31955 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 22:56:39.020057 31955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786143361779964-25976-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786143361779964-25976-0/minicluster-data/ts-1/wal
uuid: "407469795bc444cd9ff14296135e0ded"
format_stamp: "Formatted at 2026-08-07 22:56:39 on dist-test-slave-z8x9"
server_key: "120f42a5c29e8baa5b84745bfe806501"
server_key_iv: "7e396d9ea0a052f06258cc6edfeb161c"
server_key_version: "encryptionkey@0"
I20260807 22:56:39.020247 31955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786143361779964-25976-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786143361779964-25976-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786143361779964-25976-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:56:39.047992 31955 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:56:39.052529 31955 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:56:39.052870 31955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:56:39.053977 31955 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:56:39.055455 31955 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:56:39.055533 31955 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:56:39.055665 31955 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:56:39.055717 31955 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:56:39.092206 31955 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.2:34309
I20260807 22:56:39.092257 32141 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.2:34309 every 8 connection(s)
I20260807 22:56:39.093663 31955 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786143361779964-25976-0/minicluster-data/ts-1/data/info.pb
I20260807 22:56:39.102918 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 31955
I20260807 22:56:39.103130 25976 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786143361779964-25976-0/minicluster-data/ts-1/wal/instance
I20260807 22:56:39.103629 25976 external_mini_cluster.cc:1428] Setting key 3825688fe8b4a18071ae5e71d4aa4f2b
Aug 07 22:56:39 dist-test-slave-z8x9 krb5kdc[31578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143398, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.2@KRBTEST.COM for kudu/127.25.94.62@KRBTEST.COM
I20260807 22:56:39.117673 32145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:39.096447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.2:35599 (local address 127.25.94.62:36195)
0807 22:56:39.096945 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:39.096950 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:39.097885 (+ 935us) server_negotiation.cc:407] Connection header received
0807 22:56:39.099804 (+ 1919us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:39.099810 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:39.099914 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:39.100041 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:56:39.101772 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:39.102580 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:39.104789 (+ 2209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:39.105496 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:39.105639 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:39.110618 (+ 4979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:56:39.110624 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:56:39.110628 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:56:39.110663 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:56:39.113828 (+ 3165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:39.114678 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:39.114686 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:39.114691 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:39.114874 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:39.115402 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:39.115410 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:39.115416 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:39.115850 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:56:39.116011 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:39.116811 (+ 800us) server_negotiation.cc:299] Negotiation successful
0807 22:56:39.117081 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":270,"threads_started":1}
I20260807 22:56:39.118377 32146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:39.096801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:36195 (local address 127.25.94.2:35599)
0807 22:56:39.097695 (+ 894us) negotiation.cc:107] Waiting for socket to connect
0807 22:56:39.097737 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0807 22:56:39.099505 (+ 1768us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:56:39.100307 (+ 802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:56:39.100316 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:56:39.100893 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:56:39.101575 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:39.101595 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:39.102776 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:39.102780 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:39.104487 (+ 1707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:39.104510 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:39.105761 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:39.105773 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:39.106062 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:39.106925 (+ 863us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:56:39.106965 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:56:39.110327 (+ 3362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:56:39.114010 (+ 3683us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:56:39.114018 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:56:39.114032 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:56:39.114521 (+ 489us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:56:39.115040 (+ 519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:56:39.115044 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:56:39.115047 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:56:39.115264 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:56:39.116008 (+ 744us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:56:39.116015 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:56:39.116388 (+ 373us) client_negotiation.cc:769] Sending connection context
0807 22:56:39.116655 (+ 267us) client_negotiation.cc:240] Negotiation successful
0807 22:56:39.117080 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":658,"thread_start_us":425,"threads_started":1}
I20260807 22:56:39.120132 32142 heartbeater.cc:347] Connected to a master server at 127.25.94.62:36195
I20260807 22:56:39.120528 32142 heartbeater.cc:464] Registering TS with master...
I20260807 22:56:39.121367 32142 heartbeater.cc:511] Master 127.25.94.62:36195 requested a full tablet report, sending...
I20260807 22:56:39.123049 31644 ts_manager.cc:194] Registered new tserver with Master: 407469795bc444cd9ff14296135e0ded (127.25.94.2:34309)
I20260807 22:56:39.123994 31644 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.94.2@KRBTEST.COM'} at 127.25.94.2:35599
WARNING: no policy specified for kudu/127.25.94.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.3.keytab.
Entry for principal kudu/127.25.94.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.3.keytab.
I20260807 22:56:39.161548 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786143361779964-25976-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786143361779964-25976-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786143361779964-25976-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786143361779964-25976-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.3:0
--local_ip_for_outbound_sockets=127.25.94.3
--webserver_interface=127.25.94.3
--webserver_port=0
--tserver_master_addrs=127.25.94.62:36195
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.3.keytab
--principal=kudu/127.25.94.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.20:32895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786143361779964-25976-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786143361779964-25976-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786143361779964-25976-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:56:39.502871 32149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:56:39.503211 32149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:56:39.503388 32149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:56:39.510957 32149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:56:39.511121 32149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:56:39.511319 32149 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.3
I20260807 22:56:39.521332 32149 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:32895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786143361779964-25976-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786143361779964-25976-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.3.keytab
--principal=kudu/127.25.94.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786143361779964-25976-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.94.3
--webserver_port=0
--tserver_master_addrs=127.25.94.62:36195
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.3
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786143361779964-25976-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 22:56:39.522747 32149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:56:39.524242 32149 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:56:39.538407 32158 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:56:39.538501 32160 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:56:39.538558 32156 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:56:39.539597 32149 server_base.cc:1034] running on GCE node
I20260807 22:56:39.540315 32149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:56:39.541452 32149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:56:39.542686 32149 hybrid_clock.cc:648] HybridClock initialized: now 1786143399542629 us; error 56 us; skew 500 ppm
I20260807 22:56:39.543016 32149 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 22:56:39 dist-test-slave-z8x9 krb5kdc[31578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143399, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:56:39.546406 32149 init.cc:377] Logged in from keytab as kudu/127.25.94.3@KRBTEST.COM (short username kudu)
I20260807 22:56:39.548034 32149 webserver.cc:460] Webserver started at http://127.25.94.3:44835/ using document root <none> and password file <none>
I20260807 22:56:39.548730 32149 fs_manager.cc:359] Metadata directory not provided
I20260807 22:56:39.548848 32149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:56:39.549114 32149 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:56:39.552160 32149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786143361779964-25976-0/minicluster-data/ts-2/data/instance:
uuid: "343c19044ff24cd7a12cea1c0cef72e5"
format_stamp: "Formatted at 2026-08-07 22:56:39 on dist-test-slave-z8x9"
server_key: "1dd8898a7e422b24d18ddaa42575af98"
server_key_iv: "8c725364141b55dcf3ff7da618333520"
server_key_version: "encryptionkey@0"
I20260807 22:56:39.552946 32149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786143361779964-25976-0/minicluster-data/ts-2/wal/instance:
uuid: "343c19044ff24cd7a12cea1c0cef72e5"
format_stamp: "Formatted at 2026-08-07 22:56:39 on dist-test-slave-z8x9"
server_key: "1dd8898a7e422b24d18ddaa42575af98"
server_key_iv: "8c725364141b55dcf3ff7da618333520"
server_key_version: "encryptionkey@0"
I20260807 22:56:39.558531 32149 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260807 22:56:39.562263 32171 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:56:39.563841 32149 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260807 22:56:39.563988 32149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786143361779964-25976-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786143361779964-25976-0/minicluster-data/ts-2/wal
uuid: "343c19044ff24cd7a12cea1c0cef72e5"
format_stamp: "Formatted at 2026-08-07 22:56:39 on dist-test-slave-z8x9"
server_key: "1dd8898a7e422b24d18ddaa42575af98"
server_key_iv: "8c725364141b55dcf3ff7da618333520"
server_key_version: "encryptionkey@0"
I20260807 22:56:39.564174 32149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786143361779964-25976-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786143361779964-25976-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786143361779964-25976-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:56:39.589133 32149 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:56:39.592029 31948 heartbeater.cc:503] Master 127.25.94.62:36195 was elected leader, sending a full tablet report...
I20260807 22:56:39.594221 32149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:56:39.594563 32149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:56:39.595647 32149 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:56:39.597077 32149 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:56:39.597223 32149 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:56:39.597425 32149 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:56:39.597481 32149 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:56:39.632998 32149 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.3:32797
I20260807 22:56:39.633093 32342 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.3:32797 every 8 connection(s)
I20260807 22:56:39.634457 32149 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786143361779964-25976-0/minicluster-data/ts-2/data/info.pb
I20260807 22:56:39.635085 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 32149
I20260807 22:56:39.635305 25976 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786143361779964-25976-0/minicluster-data/ts-2/wal/instance
I20260807 22:56:39.635871 25976 external_mini_cluster.cc:1428] Setting key 37f2a3a05468010efba7f08e0f5f85b2
Aug 07 22:56:39 dist-test-slave-z8x9 krb5kdc[31578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143399, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.3@KRBTEST.COM for kudu/127.25.94.62@KRBTEST.COM
I20260807 22:56:39.655499 32346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:39.637511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.3:44387 (local address 127.25.94.62:36195)
0807 22:56:39.638019 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:39.638023 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:39.639328 (+ 1305us) server_negotiation.cc:407] Connection header received
0807 22:56:39.641223 (+ 1895us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:39.641226 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:39.641310 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:39.641400 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:56:39.642828 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:39.643637 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:39.645122 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:39.645562 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:39.645656 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:39.649092 (+ 3436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:56:39.649097 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:56:39.649100 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:56:39.649130 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:56:39.652288 (+ 3158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:39.653337 (+ 1049us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:39.653341 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:39.653344 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:39.653475 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:39.653901 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:39.653904 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:39.653906 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:39.654161 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:56:39.654272 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:39.655015 (+ 743us) server_negotiation.cc:299] Negotiation successful
0807 22:56:39.655196 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":244,"threads_started":1}
I20260807 22:56:39.656499 32347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:39.638225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:36195 (local address 127.25.94.3:44387)
0807 22:56:39.639119 (+ 894us) negotiation.cc:107] Waiting for socket to connect
0807 22:56:39.639191 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0807 22:56:39.640949 (+ 1758us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:56:39.641612 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:56:39.641619 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:56:39.642136 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:56:39.642676 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:39.642692 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:39.643801 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:39.643805 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:39.645004 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:39.645016 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:39.645744 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:39.645749 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:39.645940 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:39.646687 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:56:39.646710 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:56:39.648857 (+ 2147us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:56:39.652430 (+ 3573us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:56:39.652437 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:56:39.652450 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:56:39.653101 (+ 651us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:56:39.653586 (+ 485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:56:39.653590 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:56:39.653592 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:56:39.653802 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:56:39.654318 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:56:39.654324 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:56:39.654667 (+ 343us) client_negotiation.cc:769] Sending connection context
0807 22:56:39.654925 (+ 258us) client_negotiation.cc:240] Negotiation successful
0807 22:56:39.655360 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":345,"threads_started":1}
I20260807 22:56:39.658030 32343 heartbeater.cc:347] Connected to a master server at 127.25.94.62:36195
I20260807 22:56:39.658353 32343 heartbeater.cc:464] Registering TS with master...
I20260807 22:56:39.659088 32343 heartbeater.cc:511] Master 127.25.94.62:36195 requested a full tablet report, sending...
I20260807 22:56:39.660679 31644 ts_manager.cc:194] Registered new tserver with Master: 343c19044ff24cd7a12cea1c0cef72e5 (127.25.94.3:32797)
I20260807 22:56:39.661608 31644 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.94.3@KRBTEST.COM'} at 127.25.94.3:44387
I20260807 22:56:39.664242 25976 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 07 22:56:39 dist-test-slave-z8x9 krb5kdc[31578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143399, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 07 22:56:39 dist-test-slave-z8x9 krb5kdc[31578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143399, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.1@KRBTEST.COM
I20260807 22:56:39.718737 32353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:39.702952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55582 (local address 127.25.94.1:35043)
0807 22:56:39.703636 (+ 684us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:39.703647 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:39.703679 (+ 32us) server_negotiation.cc:407] Connection header received
0807 22:56:39.703912 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:39.703920 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:39.704133 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:39.704310 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:56:39.705906 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:39.706970 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:39.708203 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:39.708772 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:39.708917 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:39.712923 (+ 4006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:56:39.712935 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:56:39.712948 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:56:39.712985 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:56:39.715925 (+ 2940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:39.716615 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:39.716621 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:39.716624 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:39.716752 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:39.717153 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:39.717158 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:39.717160 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:39.717555 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:56:39.717780 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:39.718110 (+ 330us) server_negotiation.cc:299] Negotiation successful
0807 22:56:39.718306 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":506,"thread_start_us":308,"threads_started":1}
Aug 07 22:56:39 dist-test-slave-z8x9 krb5kdc[31578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143399, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.62@KRBTEST.COM
I20260807 22:56:39.738301 32346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:39.723690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52000 (local address 127.25.94.62:36195)
0807 22:56:39.723911 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:39.723919 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:39.724053 (+ 134us) server_negotiation.cc:407] Connection header received
0807 22:56:39.724177 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:39.724181 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:39.724259 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:39.724344 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:56:39.725835 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:39.726615 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:39.727878 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:39.728280 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:39.728461 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:39.732627 (+ 4166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:56:39.732633 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:56:39.732636 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:56:39.732676 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:56:39.735618 (+ 2942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:39.736351 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:39.736355 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:39.736358 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:39.736516 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:39.736921 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:39.736924 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:39.736926 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:39.737331 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:56:39.737534 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:39.737882 (+ 348us) server_negotiation.cc:299] Negotiation successful
0807 22:56:39.738073 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260807 22:56:39.739054 31644 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52000
I20260807 22:56:39.740141 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 31735
I20260807 22:56:39.753616 32353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:39.741213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55586 (local address 127.25.94.1:35043)
0807 22:56:39.741433 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:39.741441 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:39.741636 (+ 195us) server_negotiation.cc:407] Connection header received
0807 22:56:39.741839 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:39.741843 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:39.741906 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:39.742011 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:56:39.743240 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:39.743944 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:39.745187 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:39.745623 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:39.745769 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:39.748109 (+ 2340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:56:39.748114 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:56:39.748117 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:56:39.748148 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:56:39.750970 (+ 2822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:39.751625 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:39.751632 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:39.751635 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:39.751780 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:39.752157 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:39.752162 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:39.752165 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:39.752421 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:56:39.752674 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:39.753007 (+ 333us) server_negotiation.cc:299] Negotiation successful
0807 22:56:39.753294 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260807 22:56:39.754279 31939 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 22:56:39.838043 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 31735
I20260807 22:56:39.860479 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 31955
Aug 07 22:56:39 dist-test-slave-z8x9 krb5kdc[31578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143399, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.2@KRBTEST.COM
I20260807 22:56:39.877291 32361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:39.861519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49494 (local address 127.25.94.2:34309)
0807 22:56:39.862118 (+ 599us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:39.862125 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:39.862152 (+ 27us) server_negotiation.cc:407] Connection header received
0807 22:56:39.862214 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:39.862220 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:39.862440 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:39.862577 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:56:39.863763 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:39.864802 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:39.866214 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:39.866822 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:39.866925 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:39.871513 (+ 4588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:56:39.871526 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:56:39.871540 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:56:39.871571 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:56:39.874337 (+ 2766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:39.875014 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:39.875021 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:39.875024 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:39.875148 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:39.875621 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:39.875625 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:39.875627 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:39.875957 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:56:39.876291 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:39.876602 (+ 311us) server_negotiation.cc:299] Negotiation successful
0807 22:56:39.876871 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":440,"thread_start_us":328,"threads_started":1}
I20260807 22:56:39.878620 32135 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 22:56:39.962903 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 31955
I20260807 22:56:39.989871 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 32149
Aug 07 22:56:39 dist-test-slave-z8x9 krb5kdc[31578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143399, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.3@KRBTEST.COM
I20260807 22:56:40.007721 32365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:39.991014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57436 (local address 127.25.94.3:32797)
0807 22:56:39.991695 (+ 681us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:39.991703 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:39.991737 (+ 34us) server_negotiation.cc:407] Connection header received
0807 22:56:39.991800 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:39.991806 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:39.992031 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:39.992163 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:56:39.993701 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:39.994832 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:39.996267 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:39.996866 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:39.996972 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:40.001885 (+ 4913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:56:40.001900 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:56:40.001916 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:56:40.001962 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:56:40.004863 (+ 2901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:40.005647 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:40.005656 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:40.005658 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:40.005806 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:40.006169 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:40.006173 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:40.006175 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:40.006577 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:56:40.006835 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:40.007087 (+ 252us) server_negotiation.cc:299] Negotiation successful
0807 22:56:40.007382 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":526,"thread_start_us":369,"threads_started":1}
I20260807 22:56:40.009250 32336 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 22:56:40.092238 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 32149
I20260807 22:56:40.117698 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 31597
I20260807 22:56:40.119124 31685 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 22:56:40.226770 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 31597
2026-08-07T22:56:40Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (4352 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786143361779964-25976-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 22:56:40 dist-test-slave-z8x9 krb5kdc[32372](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 22:56:40 dist-test-slave-z8x9 krb5kdc[32372](info): set up 2 sockets
Aug 07 22:56:40 dist-test-slave-z8x9 krb5kdc[32372](info): commencing operation
krb5kdc: starting...
W20260807 22:56:41.582065 25976 mini_kdc.cc:121] Time spent starting KDC: real 1.301s user 0.004s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786143361779964-25976-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786143361779964-25976-0/krb5kdc/test-user.keytab.
Aug 07 22:56:41 dist-test-slave-z8x9 krb5kdc[32372](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143401, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T22:56:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T22:56:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.94.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.62.keytab.
Entry for principal kudu/127.25.94.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.62.keytab.
I20260807 22:56:41.812812 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786143361779964-25976-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786143361779964-25976-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786143361779964-25976-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.62:36407
--webserver_interface=127.25.94.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.62.keytab
--principal=kudu/127.25.94.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.20:37941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.94.62:36407
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786143361779964-25976-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786143361779964-25976-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786143361779964-25976-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:56:42.151269 32393 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:56:42.151610 32393 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:56:42.151786 32393 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:56:42.159391 32393 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 22:56:42.159529 32393 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:56:42.159603 32393 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:56:42.159693 32393 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 22:56:42.159780 32393 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 22:56:42.170186 32393 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:37941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786143361779964-25976-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786143361779964-25976-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.94.62:36407
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.62.keytab
--principal=kudu/127.25.94.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.62:36407
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.94.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786143361779964-25976-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 22:56:42.171625 32393 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:56:42.173058 32393 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:56:42.185314 32402 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:56:42.185413 32401 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:56:42.185405 32407 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:56:42.186523 32393 server_base.cc:1034] running on GCE node
I20260807 22:56:42.187358 32393 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:56:42.188845 32393 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:56:42.190078 32393 hybrid_clock.cc:648] HybridClock initialized: now 1786143402190038 us; error 37 us; skew 500 ppm
I20260807 22:56:42.190481 32393 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 22:56:42 dist-test-slave-z8x9 krb5kdc[32372](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143402, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:56:42.194092 32393 init.cc:377] Logged in from keytab as kudu/127.25.94.62@KRBTEST.COM (short username kudu)
I20260807 22:56:42.195588 32393 fs_manager.cc:359] Metadata directory not provided
I20260807 22:56:42.195747 32393 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:56:42.196120 32393 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:56:42.199792 32393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786143361779964-25976-0/minicluster-data/master-0/data/instance:
uuid: "c9b580662bba4eafaf0473020b226f60"
format_stamp: "Formatted at 2026-08-07 22:56:42 on dist-test-slave-z8x9"
server_key: "ced73df66a8386d95ccfc35f2a61a1ba"
server_key_iv: "65185b874cfbc25b79130a0045787748"
server_key_version: "encryptionkey@0"
I20260807 22:56:42.200747 32393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786143361779964-25976-0/minicluster-data/master-0/wal/instance:
uuid: "c9b580662bba4eafaf0473020b226f60"
format_stamp: "Formatted at 2026-08-07 22:56:42 on dist-test-slave-z8x9"
server_key: "ced73df66a8386d95ccfc35f2a61a1ba"
server_key_iv: "65185b874cfbc25b79130a0045787748"
server_key_version: "encryptionkey@0"
I20260807 22:56:42.206378 32393 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260807 22:56:42.210175 32415 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:56:42.211861 32393 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 22:56:42.212033 32393 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786143361779964-25976-0/minicluster-data/master-0/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786143361779964-25976-0/minicluster-data/master-0/wal
uuid: "c9b580662bba4eafaf0473020b226f60"
format_stamp: "Formatted at 2026-08-07 22:56:42 on dist-test-slave-z8x9"
server_key: "ced73df66a8386d95ccfc35f2a61a1ba"
server_key_iv: "65185b874cfbc25b79130a0045787748"
server_key_version: "encryptionkey@0"
I20260807 22:56:42.212378 32393 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786143361779964-25976-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786143361779964-25976-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786143361779964-25976-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:56:42.246301 32393 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:56:42.251324 32393 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:56:42.251581 32393 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:56:42.271641 32393 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.62:36407
I20260807 22:56:42.271684 32498 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.62:36407 every 8 connection(s)
I20260807 22:56:42.272639 32393 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
I20260807 22:56:42.276584 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 32393
I20260807 22:56:42.276691 32500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:56:42.276844 25976 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786143361779964-25976-0/minicluster-data/master-0/wal/instance
I20260807 22:56:42.277793 25976 external_mini_cluster.cc:1428] Setting key e4fd17dc40a9acf376e5e975004b8b90
I20260807 22:56:42.291517 32500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9b580662bba4eafaf0473020b226f60: Bootstrap starting.
Aug 07 22:56:42 dist-test-slave-z8x9 krb5kdc[32372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143401, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.62@KRBTEST.COM
I20260807 22:56:42.295428 32500 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c9b580662bba4eafaf0473020b226f60: Neither blocks nor log segments found. Creating new log.
I20260807 22:56:42.296567 32500 log.cc:824] T 00000000000000000000000000000000 P c9b580662bba4eafaf0473020b226f60: Log is configured to *not* fsync() on all Append() calls
I20260807 22:56:42.300025 32500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9b580662bba4eafaf0473020b226f60: No bootstrap required, opened a new log
I20260807 22:56:42.303038 32505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:42.280051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51616 (local address 127.25.94.62:36407)
0807 22:56:42.281024 (+ 973us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:42.281035 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:42.281078 (+ 43us) server_negotiation.cc:407] Connection header received
0807 22:56:42.283423 (+ 2345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:42.283449 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:42.283887 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:42.284243 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:56:42.286206 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:42.287528 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:42.289351 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:42.290029 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:42.290177 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:42.295083 (+ 4906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:56:42.295091 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:56:42.295107 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:56:42.295154 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:56:42.298602 (+ 3448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:42.299386 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:42.299394 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:42.299401 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:42.299608 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:42.300025 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:42.300029 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:42.300031 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:42.300651 (+ 620us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:56:42.300982 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:42.301459 (+ 477us) server_negotiation.cc:299] Negotiation successful
0807 22:56:42.301941 (+ 482us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":737,"thread_start_us":487,"threads_started":1}
I20260807 22:56:42.308348 32500 raft_consensus.cc:348] T 00000000000000000000000000000000 P c9b580662bba4eafaf0473020b226f60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9b580662bba4eafaf0473020b226f60" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 36407 } }
I20260807 22:56:42.308730 32500 raft_consensus.cc:374] T 00000000000000000000000000000000 P c9b580662bba4eafaf0473020b226f60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:56:42.308853 32500 raft_consensus.cc:729] T 00000000000000000000000000000000 P c9b580662bba4eafaf0473020b226f60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9b580662bba4eafaf0473020b226f60, State: Initialized, Role: FOLLOWER
I20260807 22:56:42.309489 32500 consensus_queue.cc:260] T 00000000000000000000000000000000 P c9b580662bba4eafaf0473020b226f60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9b580662bba4eafaf0473020b226f60" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 36407 } }
I20260807 22:56:42.309732 32500 raft_consensus.cc:388] T 00000000000000000000000000000000 P c9b580662bba4eafaf0473020b226f60 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 22:56:42.309863 32500 raft_consensus.cc:482] T 00000000000000000000000000000000 P c9b580662bba4eafaf0473020b226f60 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 22:56:42.310035 32500 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c9b580662bba4eafaf0473020b226f60 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:56:42.312636 32500 raft_consensus.cc:504] T 00000000000000000000000000000000 P c9b580662bba4eafaf0473020b226f60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9b580662bba4eafaf0473020b226f60" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 36407 } }
I20260807 22:56:42.313328 32500 leader_election.cc:302] T 00000000000000000000000000000000 P c9b580662bba4eafaf0473020b226f60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c9b580662bba4eafaf0473020b226f60; no voters:
I20260807 22:56:42.314028 32500 leader_election.cc:288] T 00000000000000000000000000000000 P c9b580662bba4eafaf0473020b226f60 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 22:56:42.314165 32507 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c9b580662bba4eafaf0473020b226f60 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:56:42.314924 32507 raft_consensus.cc:686] T 00000000000000000000000000000000 P c9b580662bba4eafaf0473020b226f60 [term 1 LEADER]: Becoming Leader. State: Replica: c9b580662bba4eafaf0473020b226f60, State: Running, Role: LEADER
I20260807 22:56:42.315616 32507 consensus_queue.cc:237] T 00000000000000000000000000000000 P c9b580662bba4eafaf0473020b226f60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9b580662bba4eafaf0473020b226f60" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 36407 } }
I20260807 22:56:42.316565 32500 sys_catalog.cc:564] T 00000000000000000000000000000000 P c9b580662bba4eafaf0473020b226f60 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 22:56:42.318920 32509 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9b580662bba4eafaf0473020b226f60 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c9b580662bba4eafaf0473020b226f60" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9b580662bba4eafaf0473020b226f60" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 36407 } } }
I20260807 22:56:42.319427 32509 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9b580662bba4eafaf0473020b226f60 [sys.catalog]: This master's current role is: LEADER
I20260807 22:56:42.318921 32517 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9b580662bba4eafaf0473020b226f60 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c9b580662bba4eafaf0473020b226f60. Latest consensus state: current_term: 1 leader_uuid: "c9b580662bba4eafaf0473020b226f60" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9b580662bba4eafaf0473020b226f60" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 36407 } } }
I20260807 22:56:42.319828 32517 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9b580662bba4eafaf0473020b226f60 [sys.catalog]: This master's current role is: LEADER
I20260807 22:56:42.321512 32522 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 22:56:42.326059 32522 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 22:56:42.335547 32522 catalog_manager.cc:1347] Generated new cluster ID: f64cc540338241f1af2525cec338f823
I20260807 22:56:42.335702 32522 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 22:56:42.352730 32522 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 22:56:42.354192 32522 catalog_manager.cc:1504] Loading token signing keys...
I20260807 22:56:42.361397 32522 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c9b580662bba4eafaf0473020b226f60: Generated new TSK 0
I20260807 22:56:42.362326 32522 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.94.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.1.keytab.
Entry for principal kudu/127.25.94.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.1.keytab.
I20260807 22:56:42.431350 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786143361779964-25976-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786143361779964-25976-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786143361779964-25976-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786143361779964-25976-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.1:0
--local_ip_for_outbound_sockets=127.25.94.1
--webserver_interface=127.25.94.1
--webserver_port=0
--tserver_master_addrs=127.25.94.62:36407
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.1.keytab
--principal=kudu/127.25.94.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.20:37941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786143361779964-25976-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786143361779964-25976-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786143361779964-25976-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:56:42.773758 32554 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:56:42.774066 32554 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:56:42.774243 32554 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:56:42.781883 32554 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:56:42.782004 32554 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:56:42.782179 32554 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.1
I20260807 22:56:42.792075 32554 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:37941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786143361779964-25976-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786143361779964-25976-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.1.keytab
--principal=kudu/127.25.94.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786143361779964-25976-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.94.1
--webserver_port=0
--tserver_master_addrs=127.25.94.62:36407
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.1
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786143361779964-25976-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 22:56:42.793511 32554 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:56:42.794821 32554 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:56:42.808029 32561 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:56:42.807976 32564 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:56:42.808061 32567 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:56:42.808974 32554 server_base.cc:1034] running on GCE node
I20260807 22:56:42.809621 32554 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:56:42.810635 32554 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:56:42.811844 32554 hybrid_clock.cc:648] HybridClock initialized: now 1786143402811774 us; error 67 us; skew 500 ppm
I20260807 22:56:42.812196 32554 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 22:56:42 dist-test-slave-z8x9 krb5kdc[32372](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143402, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:56:42.815553 32554 init.cc:377] Logged in from keytab as kudu/127.25.94.1@KRBTEST.COM (short username kudu)
I20260807 22:56:42.816869 32554 fs_manager.cc:359] Metadata directory not provided
I20260807 22:56:42.817001 32554 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:56:42.817440 32554 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:56:42.820611 32554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786143361779964-25976-0/minicluster-data/ts-0/data/instance:
uuid: "007c50364bbf48d9834df4c00fa25f33"
format_stamp: "Formatted at 2026-08-07 22:56:42 on dist-test-slave-z8x9"
server_key: "ebe94a345e3bb6b15d88b3fc66512467"
server_key_iv: "676046d8d6b91d7d9ff0e44402cb8ca0"
server_key_version: "encryptionkey@0"
I20260807 22:56:42.821578 32554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786143361779964-25976-0/minicluster-data/ts-0/wal/instance:
uuid: "007c50364bbf48d9834df4c00fa25f33"
format_stamp: "Formatted at 2026-08-07 22:56:42 on dist-test-slave-z8x9"
server_key: "ebe94a345e3bb6b15d88b3fc66512467"
server_key_iv: "676046d8d6b91d7d9ff0e44402cb8ca0"
server_key_version: "encryptionkey@0"
I20260807 22:56:42.828015 32554 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260807 22:56:42.831753 32576 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:56:42.833360 32554 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260807 22:56:42.833571 32554 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786143361779964-25976-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786143361779964-25976-0/minicluster-data/ts-0/wal
uuid: "007c50364bbf48d9834df4c00fa25f33"
format_stamp: "Formatted at 2026-08-07 22:56:42 on dist-test-slave-z8x9"
server_key: "ebe94a345e3bb6b15d88b3fc66512467"
server_key_iv: "676046d8d6b91d7d9ff0e44402cb8ca0"
server_key_version: "encryptionkey@0"
I20260807 22:56:42.833891 32554 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786143361779964-25976-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786143361779964-25976-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786143361779964-25976-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:56:42.858139 32554 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:56:42.866721 32554 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:56:42.867048 32554 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:56:42.868058 32554 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:56:42.869483 32554 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:56:42.869617 32554 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:56:42.869742 32554 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:56:42.869830 32554 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:56:42.904196 32554 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.1:43585
I20260807 22:56:42.904299 32745 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.1:43585 every 8 connection(s)
I20260807 22:56:42.905136 32554 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786143361779964-25976-0/minicluster-data/ts-0/data/info.pb
I20260807 22:56:42.915390 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 32554
I20260807 22:56:42.915723 25976 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786143361779964-25976-0/minicluster-data/ts-0/wal/instance
I20260807 22:56:42.916908 25976 external_mini_cluster.cc:1428] Setting key c1c3601e74119c9b77a299d64c7b0e4d
Aug 07 22:56:42 dist-test-slave-z8x9 krb5kdc[32372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143402, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.1@KRBTEST.COM for kudu/127.25.94.62@KRBTEST.COM
I20260807 22:56:42.927778 32748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:42.907869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.1:56419 (local address 127.25.94.62:36407)
0807 22:56:42.908359 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:42.908364 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:42.909112 (+ 748us) server_negotiation.cc:407] Connection header received
0807 22:56:42.911046 (+ 1934us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:42.911052 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:42.911147 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:42.911265 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:56:42.912917 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:42.913702 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:42.915271 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:42.915729 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:42.915875 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:42.920464 (+ 4589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:56:42.920473 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:56:42.920481 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:56:42.920530 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:56:42.924309 (+ 3779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:42.925239 (+ 930us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:42.925247 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:42.925253 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:42.925425 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:42.925869 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:42.925876 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:42.925882 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:42.926311 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:56:42.926466 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:42.927301 (+ 835us) server_negotiation.cc:299] Negotiation successful
0807 22:56:42.927515 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":271,"threads_started":1}
I20260807 22:56:42.929423 32749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:42.908225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:36407 (local address 127.25.94.1:56419)
0807 22:56:42.908934 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0807 22:56:42.908978 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0807 22:56:42.910757 (+ 1779us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:56:42.911502 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:56:42.911511 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:56:42.912058 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:56:42.912744 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:42.912763 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:42.913866 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:42.913872 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:42.915113 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:42.915126 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:42.916069 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:42.916077 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:42.916646 (+ 569us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:42.917501 (+ 855us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:56:42.917533 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:56:42.920204 (+ 2671us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:56:42.924493 (+ 4289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:56:42.924501 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:56:42.924522 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:56:42.925094 (+ 572us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:56:42.925571 (+ 477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:56:42.925575 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:56:42.925577 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:56:42.925771 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:56:42.926475 (+ 704us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:56:42.926482 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:56:42.926877 (+ 395us) client_negotiation.cc:769] Sending connection context
0807 22:56:42.927142 (+ 265us) client_negotiation.cc:240] Negotiation successful
0807 22:56:42.927821 (+ 679us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":292,"threads_started":1}
I20260807 22:56:42.931088 32746 heartbeater.cc:347] Connected to a master server at 127.25.94.62:36407
I20260807 22:56:42.931423 32746 heartbeater.cc:464] Registering TS with master...
I20260807 22:56:42.932094 32746 heartbeater.cc:511] Master 127.25.94.62:36407 requested a full tablet report, sending...
I20260807 22:56:42.934293 32441 ts_manager.cc:194] Registered new tserver with Master: 007c50364bbf48d9834df4c00fa25f33 (127.25.94.1:43585)
I20260807 22:56:42.935804 32441 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.94.1@KRBTEST.COM'} at 127.25.94.1:56419
WARNING: no policy specified for kudu/127.25.94.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.2.keytab.
Entry for principal kudu/127.25.94.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.2.keytab.
I20260807 22:56:42.969110 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786143361779964-25976-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786143361779964-25976-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786143361779964-25976-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786143361779964-25976-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.2:0
--local_ip_for_outbound_sockets=127.25.94.2
--webserver_interface=127.25.94.2
--webserver_port=0
--tserver_master_addrs=127.25.94.62:36407
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.2.keytab
--principal=kudu/127.25.94.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.20:37941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786143361779964-25976-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786143361779964-25976-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786143361779964-25976-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:56:43.326362 32752 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:56:43.326689 32752 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:56:43.326799 32752 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:56:43.334581 32752 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:56:43.334730 32752 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:56:43.334856 32752 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.2
I20260807 22:56:43.345916 32752 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:37941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786143361779964-25976-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786143361779964-25976-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.2.keytab
--principal=kudu/127.25.94.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786143361779964-25976-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.94.2
--webserver_port=0
--tserver_master_addrs=127.25.94.62:36407
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.2
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786143361779964-25976-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 22:56:43.347335 32752 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:56:43.348696 32752 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:56:43.363389 32760 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:56:43.363482 32761 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:56:43.363415 32764 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:56:43.364557 32752 server_base.cc:1034] running on GCE node
I20260807 22:56:43.365363 32752 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:56:43.366432 32752 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:56:43.367709 32752 hybrid_clock.cc:648] HybridClock initialized: now 1786143403367654 us; error 43 us; skew 500 ppm
I20260807 22:56:43.368122 32752 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 22:56:43 dist-test-slave-z8x9 krb5kdc[32372](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143403, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:56:43.371573 32752 init.cc:377] Logged in from keytab as kudu/127.25.94.2@KRBTEST.COM (short username kudu)
I20260807 22:56:43.372825 32752 fs_manager.cc:359] Metadata directory not provided
I20260807 22:56:43.372933 32752 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:56:43.373281 32752 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:56:43.376427 32752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786143361779964-25976-0/minicluster-data/ts-1/data/instance:
uuid: "a2aa384d454648d1b5a32985fa30c5b0"
format_stamp: "Formatted at 2026-08-07 22:56:43 on dist-test-slave-z8x9"
server_key: "25b689313741fc133d1f6208dadf5cac"
server_key_iv: "50cd197b9223d10406906c65536ef390"
server_key_version: "encryptionkey@0"
I20260807 22:56:43.377298 32752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786143361779964-25976-0/minicluster-data/ts-1/wal/instance:
uuid: "a2aa384d454648d1b5a32985fa30c5b0"
format_stamp: "Formatted at 2026-08-07 22:56:43 on dist-test-slave-z8x9"
server_key: "25b689313741fc133d1f6208dadf5cac"
server_key_iv: "50cd197b9223d10406906c65536ef390"
server_key_version: "encryptionkey@0"
I20260807 22:56:43.382596 32752 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260807 22:56:43.386159 308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:56:43.387620 32752 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 22:56:43.387785 32752 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786143361779964-25976-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786143361779964-25976-0/minicluster-data/ts-1/wal
uuid: "a2aa384d454648d1b5a32985fa30c5b0"
format_stamp: "Formatted at 2026-08-07 22:56:43 on dist-test-slave-z8x9"
server_key: "25b689313741fc133d1f6208dadf5cac"
server_key_iv: "50cd197b9223d10406906c65536ef390"
server_key_version: "encryptionkey@0"
I20260807 22:56:43.387979 32752 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786143361779964-25976-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786143361779964-25976-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786143361779964-25976-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:56:43.411866 32752 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:56:43.416029 32752 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:56:43.416256 32752 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:56:43.417459 32752 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:56:43.418938 32752 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:56:43.419029 32752 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:56:43.419109 32752 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:56:43.419163 32752 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:56:43.454725 32752 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.2:44463
I20260807 22:56:43.454777 488 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.2:44463 every 8 connection(s)
I20260807 22:56:43.455626 32752 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786143361779964-25976-0/minicluster-data/ts-1/data/info.pb
I20260807 22:56:43.463224 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 32752
I20260807 22:56:43.463457 25976 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786143361779964-25976-0/minicluster-data/ts-1/wal/instance
I20260807 22:56:43.463912 25976 external_mini_cluster.cc:1428] Setting key 0f9ca31b1d6bd63917354822f0f57686
Aug 07 22:56:43 dist-test-slave-z8x9 krb5kdc[32372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143403, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.2@KRBTEST.COM for kudu/127.25.94.62@KRBTEST.COM
I20260807 22:56:43.477116 493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:43.458237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.2:53063 (local address 127.25.94.62:36407)
0807 22:56:43.458716 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:43.458721 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:43.459537 (+ 816us) server_negotiation.cc:407] Connection header received
0807 22:56:43.461440 (+ 1903us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:43.461446 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:43.461540 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:43.461664 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:56:43.463217 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:43.464059 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:43.465879 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:43.466477 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:43.466612 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:43.470697 (+ 4085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:56:43.470702 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:56:43.470706 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:56:43.470743 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:56:43.473680 (+ 2937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:43.474432 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:43.474439 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:43.474444 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:43.474611 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:43.475078 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:43.475086 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:43.475092 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:43.475478 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:56:43.475634 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:43.476342 (+ 708us) server_negotiation.cc:299] Negotiation successful
0807 22:56:43.476601 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":251,"threads_started":1}
I20260807 22:56:43.477793 495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:43.458653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:36407 (local address 127.25.94.2:53063)
0807 22:56:43.459372 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0807 22:56:43.459419 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0807 22:56:43.461099 (+ 1680us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:56:43.461898 (+ 799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:56:43.461907 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:56:43.462444 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:56:43.463025 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:43.463043 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:43.464227 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:43.464249 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:43.465717 (+ 1468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:43.465730 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:43.466697 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:43.466702 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:43.466901 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:43.467649 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:56:43.467671 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:56:43.470447 (+ 2776us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:56:43.473822 (+ 3375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:56:43.473829 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:56:43.473843 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:56:43.474288 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:56:43.474751 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:56:43.474755 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:56:43.474758 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:56:43.474949 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:56:43.475610 (+ 661us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:56:43.475616 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:56:43.475941 (+ 325us) client_negotiation.cc:769] Sending connection context
0807 22:56:43.476179 (+ 238us) client_negotiation.cc:240] Negotiation successful
0807 22:56:43.476573 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":293,"threads_started":1}
I20260807 22:56:43.479310 489 heartbeater.cc:347] Connected to a master server at 127.25.94.62:36407
I20260807 22:56:43.479669 489 heartbeater.cc:464] Registering TS with master...
I20260807 22:56:43.480576 489 heartbeater.cc:511] Master 127.25.94.62:36407 requested a full tablet report, sending...
I20260807 22:56:43.482460 32441 ts_manager.cc:194] Registered new tserver with Master: a2aa384d454648d1b5a32985fa30c5b0 (127.25.94.2:44463)
I20260807 22:56:43.483726 32441 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.94.2@KRBTEST.COM'} at 127.25.94.2:53063
WARNING: no policy specified for kudu/127.25.94.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.3.keytab.
Entry for principal kudu/127.25.94.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.3.keytab.
I20260807 22:56:43.517053 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786143361779964-25976-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786143361779964-25976-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786143361779964-25976-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786143361779964-25976-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.3:0
--local_ip_for_outbound_sockets=127.25.94.3
--webserver_interface=127.25.94.3
--webserver_port=0
--tserver_master_addrs=127.25.94.62:36407
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.3.keytab
--principal=kudu/127.25.94.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.20:37941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786143361779964-25976-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786143361779964-25976-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786143361779964-25976-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:56:43.854408 498 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:56:43.854701 498 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:56:43.854802 498 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:56:43.861981 498 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:56:43.862107 498 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:56:43.862226 498 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.3
I20260807 22:56:43.871924 498 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:37941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786143361779964-25976-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786143361779964-25976-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.3.keytab
--principal=kudu/127.25.94.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786143361779964-25976-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.94.3
--webserver_port=0
--tserver_master_addrs=127.25.94.62:36407
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.3
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786143361779964-25976-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 22:56:43.873256 498 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:56:43.874549 498 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:56:43.888054 507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:56:43.888087 508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:56:43.888054 511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:56:43.888871 498 server_base.cc:1034] running on GCE node
I20260807 22:56:43.889549 498 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:56:43.890488 498 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:56:43.891764 498 hybrid_clock.cc:648] HybridClock initialized: now 1786143403891710 us; error 54 us; skew 500 ppm
I20260807 22:56:43.892253 498 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 22:56:43 dist-test-slave-z8x9 krb5kdc[32372](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143403, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:56:43.895634 498 init.cc:377] Logged in from keytab as kudu/127.25.94.3@KRBTEST.COM (short username kudu)
I20260807 22:56:43.896785 498 fs_manager.cc:359] Metadata directory not provided
I20260807 22:56:43.896929 498 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:56:43.897414 498 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:56:43.900277 498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786143361779964-25976-0/minicluster-data/ts-2/data/instance:
uuid: "f59c39ff0a1944f09761f9657eeac47f"
format_stamp: "Formatted at 2026-08-07 22:56:43 on dist-test-slave-z8x9"
server_key: "2c45bd64cd7687db6725d9c335905659"
server_key_iv: "87f9f26fa702495b5ef9481921314a40"
server_key_version: "encryptionkey@0"
I20260807 22:56:43.901105 498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786143361779964-25976-0/minicluster-data/ts-2/wal/instance:
uuid: "f59c39ff0a1944f09761f9657eeac47f"
format_stamp: "Formatted at 2026-08-07 22:56:43 on dist-test-slave-z8x9"
server_key: "2c45bd64cd7687db6725d9c335905659"
server_key_iv: "87f9f26fa702495b5ef9481921314a40"
server_key_version: "encryptionkey@0"
I20260807 22:56:43.906315 498 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260807 22:56:43.909952 523 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:56:43.911255 498 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 22:56:43.911466 498 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786143361779964-25976-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786143361779964-25976-0/minicluster-data/ts-2/wal
uuid: "f59c39ff0a1944f09761f9657eeac47f"
format_stamp: "Formatted at 2026-08-07 22:56:43 on dist-test-slave-z8x9"
server_key: "2c45bd64cd7687db6725d9c335905659"
server_key_iv: "87f9f26fa702495b5ef9481921314a40"
server_key_version: "encryptionkey@0"
I20260807 22:56:43.911700 498 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786143361779964-25976-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786143361779964-25976-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786143361779964-25976-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:56:43.926532 498 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:56:43.930575 498 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:56:43.930862 498 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:56:43.931804 498 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:56:43.933116 498 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:56:43.933297 498 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260807 22:56:43.933432 498 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:56:43.933527 498 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:56:43.938892 32746 heartbeater.cc:503] Master 127.25.94.62:36407 was elected leader, sending a full tablet report...
I20260807 22:56:43.970016 498 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.3:38231
I20260807 22:56:43.970072 699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.3:38231 every 8 connection(s)
I20260807 22:56:43.970932 498 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786143361779964-25976-0/minicluster-data/ts-2/data/info.pb
I20260807 22:56:43.979223 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 498
I20260807 22:56:43.979499 25976 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786143361779964-25976-0/minicluster-data/ts-2/wal/instance
I20260807 22:56:43.980090 25976 external_mini_cluster.cc:1428] Setting key 066f974ee75cadf14d0ff3e91fba7c73
Aug 07 22:56:43 dist-test-slave-z8x9 krb5kdc[32372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143403, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.3@KRBTEST.COM for kudu/127.25.94.62@KRBTEST.COM
I20260807 22:56:43.992136 493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:43.973433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.3:41505 (local address 127.25.94.62:36407)
0807 22:56:43.973622 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:43.973627 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:43.974821 (+ 1194us) server_negotiation.cc:407] Connection header received
0807 22:56:43.976698 (+ 1877us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:43.976703 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:43.976817 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:43.976937 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:56:43.978609 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:43.979440 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:43.981552 (+ 2112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:43.982021 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:43.982158 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:43.986121 (+ 3963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:56:43.986127 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:56:43.986131 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:56:43.986169 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:56:43.988974 (+ 2805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:43.989834 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:43.989839 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:43.989842 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:43.989968 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:43.990373 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:43.990376 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:43.990378 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:43.990659 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:56:43.990778 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:43.991649 (+ 871us) server_negotiation.cc:299] Negotiation successful
0807 22:56:43.991882 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260807 22:56:43.993254 703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:43.973896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:36407 (local address 127.25.94.3:41505)
0807 22:56:43.974646 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0807 22:56:43.974691 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0807 22:56:43.976396 (+ 1705us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:56:43.977245 (+ 849us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:56:43.977254 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:56:43.977818 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:56:43.978405 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:43.978424 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:43.979668 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:43.979677 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:43.981425 (+ 1748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:43.981439 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:43.982298 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:43.982308 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:43.982542 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:43.983415 (+ 873us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:56:43.983458 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:56:43.985864 (+ 2406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:56:43.989120 (+ 3256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:56:43.989127 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:56:43.989141 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:56:43.989706 (+ 565us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:56:43.990067 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:56:43.990070 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:56:43.990072 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:56:43.990258 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:56:43.990810 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:56:43.990817 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:56:43.991169 (+ 352us) client_negotiation.cc:769] Sending connection context
0807 22:56:43.991505 (+ 336us) client_negotiation.cc:240] Negotiation successful
0807 22:56:43.991962 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":320,"threads_started":1}
I20260807 22:56:43.994900 700 heartbeater.cc:347] Connected to a master server at 127.25.94.62:36407
I20260807 22:56:43.995273 700 heartbeater.cc:464] Registering TS with master...
I20260807 22:56:43.996163 700 heartbeater.cc:511] Master 127.25.94.62:36407 requested a full tablet report, sending...
I20260807 22:56:43.997601 32441 ts_manager.cc:194] Registered new tserver with Master: f59c39ff0a1944f09761f9657eeac47f (127.25.94.3:38231)
I20260807 22:56:43.998410 32441 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.94.3@KRBTEST.COM'} at 127.25.94.3:41505
I20260807 22:56:44.008167 25976 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 22:56:44.026800 493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:44.013202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34880 (local address 127.25.94.62:36407)
0807 22:56:44.013410 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:44.013418 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:44.014021 (+ 603us) server_negotiation.cc:407] Connection header received
0807 22:56:44.014207 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:44.014213 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:44.014309 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:44.014427 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:56:44.015960 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:44.016858 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:44.018431 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:44.018837 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:44.018931 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:44.021301 (+ 2370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:56:44.021306 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:56:44.021309 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:56:44.021340 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:56:44.023934 (+ 2594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:44.024613 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:44.024617 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:44.024620 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:44.024759 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:44.025089 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:44.025092 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:44.025094 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:44.025553 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:56:44.025862 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:44.026184 (+ 322us) server_negotiation.cc:299] Negotiation successful
0807 22:56:44.026405 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260807 22:56:44.031257 32441 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34880:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 22:56:44.038213 32441 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 22:56:44.070576 724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:44.050593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36594 (local address 127.25.94.1:43585)
0807 22:56:44.062294 (+ 11701us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:44.062307 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:44.062351 (+ 44us) server_negotiation.cc:407] Connection header received
0807 22:56:44.062478 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:44.062485 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:44.062776 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:44.063769 (+ 993us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:56:44.063951 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:44.065592 (+ 1641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:44.067895 (+ 2303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:44.069334 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:44.069486 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:44.069739 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:44.070044 (+ 305us) server_negotiation.cc:299] Negotiation successful
0807 22:56:44.070206 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11483,"thread_start_us":11273,"threads_started":1}
I20260807 22:56:44.070581 726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:44.050333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.1:43585 (local address 127.0.0.1:36594)
0807 22:56:44.059945 (+ 9612us) negotiation.cc:107] Waiting for socket to connect
0807 22:56:44.059985 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0807 22:56:44.060168 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:56:44.063107 (+ 2939us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:56:44.063119 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:56:44.063153 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:56:44.063586 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:44.063597 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:44.065743 (+ 2146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:44.065748 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:44.067708 (+ 1960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:44.067721 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:44.069567 (+ 1846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:44.069574 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:44.069783 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:44.069833 (+ 50us) client_negotiation.cc:769] Sending connection context
0807 22:56:44.069989 (+ 156us) client_negotiation.cc:240] Negotiation successful
0807 22:56:44.070192 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9332,"thread_start_us":14291,"threads_started":1}
I20260807 22:56:44.071902 723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:44.050439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.2:44463 (local address 127.0.0.1:49920)
0807 22:56:44.061874 (+ 11435us) negotiation.cc:107] Waiting for socket to connect
0807 22:56:44.061898 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0807 22:56:44.062028 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:56:44.064183 (+ 2155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:56:44.064190 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:56:44.064215 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:56:44.064680 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:44.064693 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:44.066684 (+ 1991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:44.066689 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:44.068831 (+ 2142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:44.068846 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:44.071008 (+ 2162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:44.071021 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:44.071247 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:44.071295 (+ 48us) client_negotiation.cc:769] Sending connection context
0807 22:56:44.071378 (+ 83us) client_negotiation.cc:240] Negotiation successful
0807 22:56:44.071512 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11258,"mutex_wait_us":23,"thread_start_us":11071,"threads_started":1}
I20260807 22:56:44.072327 727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:44.051889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49920 (local address 127.25.94.2:44463)
0807 22:56:44.063445 (+ 11556us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:44.063458 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:44.063495 (+ 37us) server_negotiation.cc:407] Connection header received
0807 22:56:44.063621 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:44.063631 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:44.063929 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:44.064179 (+ 250us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:56:44.064945 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:44.066468 (+ 1523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:44.069050 (+ 2582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:44.070528 (+ 1478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:44.070669 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:44.070845 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:44.071438 (+ 593us) server_negotiation.cc:299] Negotiation successful
0807 22:56:44.071603 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11351,"thread_start_us":11233,"threads_started":1}
I20260807 22:56:44.074201 722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:44.051038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.3:38231 (local address 127.0.0.1:54342)
0807 22:56:44.064941 (+ 13903us) negotiation.cc:107] Waiting for socket to connect
0807 22:56:44.064959 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 22:56:44.065202 (+ 243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:56:44.066113 (+ 911us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:56:44.066116 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:56:44.066136 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:56:44.066461 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:44.066468 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:44.068798 (+ 2330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:44.068809 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:44.071199 (+ 2390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:44.071214 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:44.073395 (+ 2181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:44.073403 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:44.073568 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:44.073600 (+ 32us) client_negotiation.cc:769] Sending connection context
0807 22:56:44.073701 (+ 101us) client_negotiation.cc:240] Negotiation successful
0807 22:56:44.073831 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13777,"mutex_wait_us":20,"thread_start_us":8464,"threads_started":1}
I20260807 22:56:44.074327 728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:44.052335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54342 (local address 127.25.94.3:38231)
0807 22:56:44.062917 (+ 10582us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:44.062929 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:44.065069 (+ 2140us) server_negotiation.cc:407] Connection header received
0807 22:56:44.065406 (+ 337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:44.065415 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:44.065629 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:44.065807 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:56:44.066669 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:44.068610 (+ 1941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:44.071376 (+ 2766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:44.072826 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:44.072979 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:44.073247 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:44.073784 (+ 537us) server_negotiation.cc:299] Negotiation successful
0807 22:56:44.073959 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10344,"thread_start_us":10122,"threads_started":1}
I20260807 22:56:44.075428 32642 tablet_service.cc:1467] Processing CreateTablet for tablet ac3dba3bcd3e46b58aba106cdfcf37d3 (DEFAULT_TABLE table=test-table [id=145dba3947be45b78df83b9778c6b7c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:56:44.075671 377 tablet_service.cc:1467] Processing CreateTablet for tablet ac3dba3bcd3e46b58aba106cdfcf37d3 (DEFAULT_TABLE table=test-table [id=145dba3947be45b78df83b9778c6b7c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:56:44.077076 32642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac3dba3bcd3e46b58aba106cdfcf37d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:56:44.077118 377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac3dba3bcd3e46b58aba106cdfcf37d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:56:44.078433 604 tablet_service.cc:1467] Processing CreateTablet for tablet ac3dba3bcd3e46b58aba106cdfcf37d3 (DEFAULT_TABLE table=test-table [id=145dba3947be45b78df83b9778c6b7c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:56:44.079857 604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac3dba3bcd3e46b58aba106cdfcf37d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:56:44.090878 730 tablet_bootstrap.cc:492] T ac3dba3bcd3e46b58aba106cdfcf37d3 P 007c50364bbf48d9834df4c00fa25f33: Bootstrap starting.
I20260807 22:56:44.093137 733 tablet_bootstrap.cc:492] T ac3dba3bcd3e46b58aba106cdfcf37d3 P a2aa384d454648d1b5a32985fa30c5b0: Bootstrap starting.
I20260807 22:56:44.094901 735 tablet_bootstrap.cc:492] T ac3dba3bcd3e46b58aba106cdfcf37d3 P f59c39ff0a1944f09761f9657eeac47f: Bootstrap starting.
I20260807 22:56:44.095978 730 tablet_bootstrap.cc:654] T ac3dba3bcd3e46b58aba106cdfcf37d3 P 007c50364bbf48d9834df4c00fa25f33: Neither blocks nor log segments found. Creating new log.
I20260807 22:56:44.097075 733 tablet_bootstrap.cc:654] T ac3dba3bcd3e46b58aba106cdfcf37d3 P a2aa384d454648d1b5a32985fa30c5b0: Neither blocks nor log segments found. Creating new log.
I20260807 22:56:44.098243 735 tablet_bootstrap.cc:654] T ac3dba3bcd3e46b58aba106cdfcf37d3 P f59c39ff0a1944f09761f9657eeac47f: Neither blocks nor log segments found. Creating new log.
I20260807 22:56:44.098344 733 log.cc:824] T ac3dba3bcd3e46b58aba106cdfcf37d3 P a2aa384d454648d1b5a32985fa30c5b0: Log is configured to *not* fsync() on all Append() calls
I20260807 22:56:44.098336 730 log.cc:824] T ac3dba3bcd3e46b58aba106cdfcf37d3 P 007c50364bbf48d9834df4c00fa25f33: Log is configured to *not* fsync() on all Append() calls
I20260807 22:56:44.099489 735 log.cc:824] T ac3dba3bcd3e46b58aba106cdfcf37d3 P f59c39ff0a1944f09761f9657eeac47f: Log is configured to *not* fsync() on all Append() calls
I20260807 22:56:44.102094 733 tablet_bootstrap.cc:492] T ac3dba3bcd3e46b58aba106cdfcf37d3 P a2aa384d454648d1b5a32985fa30c5b0: No bootstrap required, opened a new log
I20260807 22:56:44.102097 730 tablet_bootstrap.cc:492] T ac3dba3bcd3e46b58aba106cdfcf37d3 P 007c50364bbf48d9834df4c00fa25f33: No bootstrap required, opened a new log
I20260807 22:56:44.102315 735 tablet_bootstrap.cc:492] T ac3dba3bcd3e46b58aba106cdfcf37d3 P f59c39ff0a1944f09761f9657eeac47f: No bootstrap required, opened a new log
I20260807 22:56:44.102414 733 ts_tablet_manager.cc:1397] T ac3dba3bcd3e46b58aba106cdfcf37d3 P a2aa384d454648d1b5a32985fa30c5b0: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.007s
I20260807 22:56:44.102414 730 ts_tablet_manager.cc:1397] T ac3dba3bcd3e46b58aba106cdfcf37d3 P 007c50364bbf48d9834df4c00fa25f33: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s
I20260807 22:56:44.102627 735 ts_tablet_manager.cc:1397] T ac3dba3bcd3e46b58aba106cdfcf37d3 P f59c39ff0a1944f09761f9657eeac47f: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260807 22:56:44.111172 735 raft_consensus.cc:348] T ac3dba3bcd3e46b58aba106cdfcf37d3 P f59c39ff0a1944f09761f9657eeac47f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "007c50364bbf48d9834df4c00fa25f33" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 43585 } } peers { permanent_uuid: "a2aa384d454648d1b5a32985fa30c5b0" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 44463 } } peers { permanent_uuid: "f59c39ff0a1944f09761f9657eeac47f" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 38231 } }
I20260807 22:56:44.111639 735 raft_consensus.cc:374] T ac3dba3bcd3e46b58aba106cdfcf37d3 P f59c39ff0a1944f09761f9657eeac47f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:56:44.111739 735 raft_consensus.cc:729] T ac3dba3bcd3e46b58aba106cdfcf37d3 P f59c39ff0a1944f09761f9657eeac47f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f59c39ff0a1944f09761f9657eeac47f, State: Initialized, Role: FOLLOWER
I20260807 22:56:44.112443 735 consensus_queue.cc:260] T ac3dba3bcd3e46b58aba106cdfcf37d3 P f59c39ff0a1944f09761f9657eeac47f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "007c50364bbf48d9834df4c00fa25f33" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 43585 } } peers { permanent_uuid: "a2aa384d454648d1b5a32985fa30c5b0" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 44463 } } peers { permanent_uuid: "f59c39ff0a1944f09761f9657eeac47f" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 38231 } }
I20260807 22:56:44.113686 700 heartbeater.cc:503] Master 127.25.94.62:36407 was elected leader, sending a full tablet report...
I20260807 22:56:44.113924 730 raft_consensus.cc:348] T ac3dba3bcd3e46b58aba106cdfcf37d3 P 007c50364bbf48d9834df4c00fa25f33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "007c50364bbf48d9834df4c00fa25f33" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 43585 } } peers { permanent_uuid: "a2aa384d454648d1b5a32985fa30c5b0" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 44463 } } peers { permanent_uuid: "f59c39ff0a1944f09761f9657eeac47f" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 38231 } }
I20260807 22:56:44.114428 730 raft_consensus.cc:374] T ac3dba3bcd3e46b58aba106cdfcf37d3 P 007c50364bbf48d9834df4c00fa25f33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:56:44.114500 735 ts_tablet_manager.cc:1428] T ac3dba3bcd3e46b58aba106cdfcf37d3 P f59c39ff0a1944f09761f9657eeac47f: Time spent starting tablet: real 0.012s user 0.007s sys 0.005s
I20260807 22:56:44.114529 730 raft_consensus.cc:729] T ac3dba3bcd3e46b58aba106cdfcf37d3 P 007c50364bbf48d9834df4c00fa25f33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 007c50364bbf48d9834df4c00fa25f33, State: Initialized, Role: FOLLOWER
I20260807 22:56:44.114224 733 raft_consensus.cc:348] T ac3dba3bcd3e46b58aba106cdfcf37d3 P a2aa384d454648d1b5a32985fa30c5b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "007c50364bbf48d9834df4c00fa25f33" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 43585 } } peers { permanent_uuid: "a2aa384d454648d1b5a32985fa30c5b0" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 44463 } } peers { permanent_uuid: "f59c39ff0a1944f09761f9657eeac47f" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 38231 } }
I20260807 22:56:44.114748 733 raft_consensus.cc:374] T ac3dba3bcd3e46b58aba106cdfcf37d3 P a2aa384d454648d1b5a32985fa30c5b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:56:44.114864 733 raft_consensus.cc:729] T ac3dba3bcd3e46b58aba106cdfcf37d3 P a2aa384d454648d1b5a32985fa30c5b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2aa384d454648d1b5a32985fa30c5b0, State: Initialized, Role: FOLLOWER
I20260807 22:56:44.115211 730 consensus_queue.cc:260] T ac3dba3bcd3e46b58aba106cdfcf37d3 P 007c50364bbf48d9834df4c00fa25f33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "007c50364bbf48d9834df4c00fa25f33" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 43585 } } peers { permanent_uuid: "a2aa384d454648d1b5a32985fa30c5b0" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 44463 } } peers { permanent_uuid: "f59c39ff0a1944f09761f9657eeac47f" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 38231 } }
I20260807 22:56:44.115546 733 consensus_queue.cc:260] T ac3dba3bcd3e46b58aba106cdfcf37d3 P a2aa384d454648d1b5a32985fa30c5b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "007c50364bbf48d9834df4c00fa25f33" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 43585 } } peers { permanent_uuid: "a2aa384d454648d1b5a32985fa30c5b0" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 44463 } } peers { permanent_uuid: "f59c39ff0a1944f09761f9657eeac47f" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 38231 } }
I20260807 22:56:44.120842 489 heartbeater.cc:503] Master 127.25.94.62:36407 was elected leader, sending a full tablet report...
I20260807 22:56:44.123258 733 ts_tablet_manager.cc:1428] T ac3dba3bcd3e46b58aba106cdfcf37d3 P a2aa384d454648d1b5a32985fa30c5b0: Time spent starting tablet: real 0.021s user 0.015s sys 0.001s
I20260807 22:56:44.124316 730 ts_tablet_manager.cc:1428] T ac3dba3bcd3e46b58aba106cdfcf37d3 P 007c50364bbf48d9834df4c00fa25f33: Time spent starting tablet: real 0.022s user 0.017s sys 0.004s
W20260807 22:56:44.157676 32747 tablet.cc:2369] T ac3dba3bcd3e46b58aba106cdfcf37d3 P 007c50364bbf48d9834df4c00fa25f33: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 22:56:44.207791 490 tablet.cc:2369] T ac3dba3bcd3e46b58aba106cdfcf37d3 P a2aa384d454648d1b5a32985fa30c5b0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 22:56:44.222728 701 tablet.cc:2369] T ac3dba3bcd3e46b58aba106cdfcf37d3 P f59c39ff0a1944f09761f9657eeac47f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 22:56:44.300395 740 raft_consensus.cc:482] T ac3dba3bcd3e46b58aba106cdfcf37d3 P f59c39ff0a1944f09761f9657eeac47f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 22:56:44.300760 740 raft_consensus.cc:504] T ac3dba3bcd3e46b58aba106cdfcf37d3 P f59c39ff0a1944f09761f9657eeac47f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "007c50364bbf48d9834df4c00fa25f33" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 43585 } } peers { permanent_uuid: "a2aa384d454648d1b5a32985fa30c5b0" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 44463 } } peers { permanent_uuid: "f59c39ff0a1944f09761f9657eeac47f" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 38231 } }
I20260807 22:56:44.302559 740 leader_election.cc:288] T ac3dba3bcd3e46b58aba106cdfcf37d3 P f59c39ff0a1944f09761f9657eeac47f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 007c50364bbf48d9834df4c00fa25f33 (127.25.94.1:43585), a2aa384d454648d1b5a32985fa30c5b0 (127.25.94.2:44463)
I20260807 22:56:44.309480 703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:44.302911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.1:43585 (local address 127.25.94.3:44827)
0807 22:56:44.303376 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0807 22:56:44.303404 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0807 22:56:44.303552 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:56:44.303972 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:56:44.303978 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:56:44.304007 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:56:44.304222 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:44.304234 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:44.305653 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:44.305658 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:44.307356 (+ 1698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:44.307369 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:44.308589 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:44.308596 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:44.308799 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:44.308868 (+ 69us) client_negotiation.cc:769] Sending connection context
0807 22:56:44.308965 (+ 97us) client_negotiation.cc:240] Negotiation successful
0807 22:56:44.309101 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"mutex_wait_us":53}
I20260807 22:56:44.309549 724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:44.303170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.3:44827 (local address 127.25.94.1:43585)
0807 22:56:44.303592 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:44.303597 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:44.303618 (+ 21us) server_negotiation.cc:407] Connection header received
0807 22:56:44.303700 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:44.303706 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:44.303804 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:44.303910 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:56:44.304394 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:44.305499 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:44.307518 (+ 2019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:44.308414 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:44.308522 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:44.308595 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:44.309024 (+ 429us) server_negotiation.cc:299] Negotiation successful
0807 22:56:44.309244 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277}
I20260807 22:56:44.309772 727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:44.303171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.3:48983 (local address 127.25.94.2:44463)
0807 22:56:44.303414 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:44.303422 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:44.303882 (+ 460us) server_negotiation.cc:407] Connection header received
0807 22:56:44.304045 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:44.304054 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:44.304131 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:44.304234 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:56:44.304959 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:44.305861 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:44.307858 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:44.308737 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:44.308879 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:44.308983 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:44.309363 (+ 380us) server_negotiation.cc:299] Negotiation successful
0807 22:56:44.309502 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260807 22:56:44.310277 744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:44.302911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.2:44463 (local address 127.25.94.3:48983)
0807 22:56:44.303726 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0807 22:56:44.303744 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 22:56:44.303899 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:56:44.304269 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:56:44.304272 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:56:44.304319 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:56:44.304716 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:44.304729 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:44.306016 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:44.306020 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:44.307693 (+ 1673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:44.307709 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:44.308972 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:44.308978 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:44.309157 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:44.309216 (+ 59us) client_negotiation.cc:769] Sending connection context
0807 22:56:44.309311 (+ 95us) client_negotiation.cc:240] Negotiation successful
0807 22:56:44.309444 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":683,"mutex_wait_us":65,"thread_start_us":411,"threads_started":1}
I20260807 22:56:44.310627 32675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3dba3bcd3e46b58aba106cdfcf37d3" candidate_uuid: "f59c39ff0a1944f09761f9657eeac47f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "007c50364bbf48d9834df4c00fa25f33" is_pre_election: true
I20260807 22:56:44.311324 32675 raft_consensus.cc:2454] T ac3dba3bcd3e46b58aba106cdfcf37d3 P 007c50364bbf48d9834df4c00fa25f33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f59c39ff0a1944f09761f9657eeac47f in term 0.
I20260807 22:56:44.311419 418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3dba3bcd3e46b58aba106cdfcf37d3" candidate_uuid: "f59c39ff0a1944f09761f9657eeac47f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2aa384d454648d1b5a32985fa30c5b0" is_pre_election: true
I20260807 22:56:44.311990 418 raft_consensus.cc:2454] T ac3dba3bcd3e46b58aba106cdfcf37d3 P a2aa384d454648d1b5a32985fa30c5b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f59c39ff0a1944f09761f9657eeac47f in term 0.
I20260807 22:56:44.312048 526 leader_election.cc:302] T ac3dba3bcd3e46b58aba106cdfcf37d3 P f59c39ff0a1944f09761f9657eeac47f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 007c50364bbf48d9834df4c00fa25f33, f59c39ff0a1944f09761f9657eeac47f; no voters:
I20260807 22:56:44.312569 740 raft_consensus.cc:2781] T ac3dba3bcd3e46b58aba106cdfcf37d3 P f59c39ff0a1944f09761f9657eeac47f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 22:56:44.312743 740 raft_consensus.cc:482] T ac3dba3bcd3e46b58aba106cdfcf37d3 P f59c39ff0a1944f09761f9657eeac47f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 22:56:44.312898 740 raft_consensus.cc:3037] T ac3dba3bcd3e46b58aba106cdfcf37d3 P f59c39ff0a1944f09761f9657eeac47f [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:56:44.315333 740 raft_consensus.cc:504] T ac3dba3bcd3e46b58aba106cdfcf37d3 P f59c39ff0a1944f09761f9657eeac47f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "007c50364bbf48d9834df4c00fa25f33" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 43585 } } peers { permanent_uuid: "a2aa384d454648d1b5a32985fa30c5b0" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 44463 } } peers { permanent_uuid: "f59c39ff0a1944f09761f9657eeac47f" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 38231 } }
I20260807 22:56:44.316102 740 leader_election.cc:288] T ac3dba3bcd3e46b58aba106cdfcf37d3 P f59c39ff0a1944f09761f9657eeac47f [CANDIDATE]: Term 1 election: Requested vote from peers 007c50364bbf48d9834df4c00fa25f33 (127.25.94.1:43585), a2aa384d454648d1b5a32985fa30c5b0 (127.25.94.2:44463)
I20260807 22:56:44.316557 32675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3dba3bcd3e46b58aba106cdfcf37d3" candidate_uuid: "f59c39ff0a1944f09761f9657eeac47f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "007c50364bbf48d9834df4c00fa25f33"
I20260807 22:56:44.316712 418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3dba3bcd3e46b58aba106cdfcf37d3" candidate_uuid: "f59c39ff0a1944f09761f9657eeac47f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2aa384d454648d1b5a32985fa30c5b0"
I20260807 22:56:44.316817 32675 raft_consensus.cc:3037] T ac3dba3bcd3e46b58aba106cdfcf37d3 P 007c50364bbf48d9834df4c00fa25f33 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:56:44.316938 418 raft_consensus.cc:3037] T ac3dba3bcd3e46b58aba106cdfcf37d3 P a2aa384d454648d1b5a32985fa30c5b0 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:56:44.320243 32675 raft_consensus.cc:2454] T ac3dba3bcd3e46b58aba106cdfcf37d3 P 007c50364bbf48d9834df4c00fa25f33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f59c39ff0a1944f09761f9657eeac47f in term 1.
I20260807 22:56:44.320243 418 raft_consensus.cc:2454] T ac3dba3bcd3e46b58aba106cdfcf37d3 P a2aa384d454648d1b5a32985fa30c5b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f59c39ff0a1944f09761f9657eeac47f in term 1.
I20260807 22:56:44.320713 526 leader_election.cc:302] T ac3dba3bcd3e46b58aba106cdfcf37d3 P f59c39ff0a1944f09761f9657eeac47f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 007c50364bbf48d9834df4c00fa25f33, f59c39ff0a1944f09761f9657eeac47f; no voters:
I20260807 22:56:44.321040 740 raft_consensus.cc:2781] T ac3dba3bcd3e46b58aba106cdfcf37d3 P f59c39ff0a1944f09761f9657eeac47f [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:56:44.321587 740 raft_consensus.cc:686] T ac3dba3bcd3e46b58aba106cdfcf37d3 P f59c39ff0a1944f09761f9657eeac47f [term 1 LEADER]: Becoming Leader. State: Replica: f59c39ff0a1944f09761f9657eeac47f, State: Running, Role: LEADER
I20260807 22:56:44.322197 740 consensus_queue.cc:237] T ac3dba3bcd3e46b58aba106cdfcf37d3 P f59c39ff0a1944f09761f9657eeac47f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "007c50364bbf48d9834df4c00fa25f33" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 43585 } } peers { permanent_uuid: "a2aa384d454648d1b5a32985fa30c5b0" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 44463 } } peers { permanent_uuid: "f59c39ff0a1944f09761f9657eeac47f" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 38231 } }
I20260807 22:56:44.326902 32439 catalog_manager.cc:5495] T ac3dba3bcd3e46b58aba106cdfcf37d3 P f59c39ff0a1944f09761f9657eeac47f reported cstate change: term changed from 0 to 1, leader changed from <none> to f59c39ff0a1944f09761f9657eeac47f (127.25.94.3). New cstate: current_term: 1 leader_uuid: "f59c39ff0a1944f09761f9657eeac47f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "007c50364bbf48d9834df4c00fa25f33" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 43585 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2aa384d454648d1b5a32985fa30c5b0" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 44463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f59c39ff0a1944f09761f9657eeac47f" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 38231 } health_report { overall_health: HEALTHY } } }
I20260807 22:56:44.377771 728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:44.371749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54356 (local address 127.25.94.3:38231)
0807 22:56:44.372007 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:44.372013 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:44.372034 (+ 21us) server_negotiation.cc:407] Connection header received
0807 22:56:44.372290 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:44.372294 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:44.372378 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:44.372561 (+ 183us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 22:56:44.372982 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:44.373791 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:44.375217 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:44.375627 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:44.375768 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:44.376341 (+ 573us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 22:56:44.376884 (+ 543us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 22:56:44.377024 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:44.377332 (+ 308us) server_negotiation.cc:299] Negotiation successful
0807 22:56:44.377448 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260807 22:56:44.386206 418 raft_consensus.cc:1261] T ac3dba3bcd3e46b58aba106cdfcf37d3 P a2aa384d454648d1b5a32985fa30c5b0 [term 1 FOLLOWER]: Refusing update from remote peer f59c39ff0a1944f09761f9657eeac47f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 22:56:44.386215 32675 raft_consensus.cc:1261] T ac3dba3bcd3e46b58aba106cdfcf37d3 P 007c50364bbf48d9834df4c00fa25f33 [term 1 FOLLOWER]: Refusing update from remote peer f59c39ff0a1944f09761f9657eeac47f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 22:56:44.387269 745 consensus_queue.cc:1035] T ac3dba3bcd3e46b58aba106cdfcf37d3 P f59c39ff0a1944f09761f9657eeac47f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2aa384d454648d1b5a32985fa30c5b0" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 44463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 22:56:44.387691 740 consensus_queue.cc:1035] T ac3dba3bcd3e46b58aba106cdfcf37d3 P f59c39ff0a1944f09761f9657eeac47f [LEADER]: Connected to new peer: Peer: permanent_uuid: "007c50364bbf48d9834df4c00fa25f33" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 43585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 22:56:44.416137 32441 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34880:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 22:56:44.416514 32441 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34880:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 22:56:44.420288 32441 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c9b580662bba4eafaf0473020b226f60: Sending DeleteTablet for 3 replicas of tablet ac3dba3bcd3e46b58aba106cdfcf37d3
I20260807 22:56:44.421594 32642 tablet_service.cc:1514] Processing DeleteTablet for tablet ac3dba3bcd3e46b58aba106cdfcf37d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:56:44 UTC) from {username='kudu', principal='kudu/127.25.94.62@KRBTEST.COM'} at 127.0.0.1:36594
I20260807 22:56:44.421717 377 tablet_service.cc:1514] Processing DeleteTablet for tablet ac3dba3bcd3e46b58aba106cdfcf37d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:56:44 UTC) from {username='kudu', principal='kudu/127.25.94.62@KRBTEST.COM'} at 127.0.0.1:49920
I20260807 22:56:44.422499 604 tablet_service.cc:1514] Processing DeleteTablet for tablet ac3dba3bcd3e46b58aba106cdfcf37d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:56:44 UTC) from {username='kudu', principal='kudu/127.25.94.62@KRBTEST.COM'} at 127.0.0.1:54342
I20260807 22:56:44.422601 763 tablet_replica.cc:330] T ac3dba3bcd3e46b58aba106cdfcf37d3 P 007c50364bbf48d9834df4c00fa25f33: stopping tablet replica
I20260807 22:56:44.423134 763 raft_consensus.cc:2229] T ac3dba3bcd3e46b58aba106cdfcf37d3 P 007c50364bbf48d9834df4c00fa25f33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 22:56:44.423480 766 tablet_replica.cc:330] T ac3dba3bcd3e46b58aba106cdfcf37d3 P f59c39ff0a1944f09761f9657eeac47f: stopping tablet replica
I20260807 22:56:44.423667 763 raft_consensus.cc:2258] T ac3dba3bcd3e46b58aba106cdfcf37d3 P 007c50364bbf48d9834df4c00fa25f33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 22:56:44.424000 766 raft_consensus.cc:2229] T ac3dba3bcd3e46b58aba106cdfcf37d3 P f59c39ff0a1944f09761f9657eeac47f [term 1 LEADER]: Raft consensus shutting down.
I20260807 22:56:44.424677 766 raft_consensus.cc:2258] T ac3dba3bcd3e46b58aba106cdfcf37d3 P f59c39ff0a1944f09761f9657eeac47f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 22:56:44.424779 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 32554
I20260807 22:56:44.427167 766 ts_tablet_manager.cc:1905] T ac3dba3bcd3e46b58aba106cdfcf37d3 P f59c39ff0a1944f09761f9657eeac47f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 22:56:44.429864 763 ts_tablet_manager.cc:1905] T ac3dba3bcd3e46b58aba106cdfcf37d3 P 007c50364bbf48d9834df4c00fa25f33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 22:56:44.434906 766 ts_tablet_manager.cc:1918] T ac3dba3bcd3e46b58aba106cdfcf37d3 P f59c39ff0a1944f09761f9657eeac47f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 22:56:44.435101 766 log.cc:1196] T ac3dba3bcd3e46b58aba106cdfcf37d3 P f59c39ff0a1944f09761f9657eeac47f: Deleting WAL directory at /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786143361779964-25976-0/minicluster-data/ts-2/wal/wals/ac3dba3bcd3e46b58aba106cdfcf37d3
I20260807 22:56:44.435710 766 ts_tablet_manager.cc:1939] T ac3dba3bcd3e46b58aba106cdfcf37d3 P f59c39ff0a1944f09761f9657eeac47f: Deleting consensus metadata
I20260807 22:56:44.437546 764 tablet_replica.cc:330] T ac3dba3bcd3e46b58aba106cdfcf37d3 P a2aa384d454648d1b5a32985fa30c5b0: stopping tablet replica
I20260807 22:56:44.437989 764 raft_consensus.cc:2229] T ac3dba3bcd3e46b58aba106cdfcf37d3 P a2aa384d454648d1b5a32985fa30c5b0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 22:56:44.438524 764 raft_consensus.cc:2258] T ac3dba3bcd3e46b58aba106cdfcf37d3 P a2aa384d454648d1b5a32985fa30c5b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 22:56:44.439738 763 ts_tablet_manager.cc:1918] T ac3dba3bcd3e46b58aba106cdfcf37d3 P 007c50364bbf48d9834df4c00fa25f33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 22:56:44.439751 32424 catalog_manager.cc:4841] TS f59c39ff0a1944f09761f9657eeac47f (127.25.94.3:38231): tablet ac3dba3bcd3e46b58aba106cdfcf37d3 (table test-table [id=145dba3947be45b78df83b9778c6b7c1]) successfully deleted
I20260807 22:56:44.439965 763 log.cc:1196] T ac3dba3bcd3e46b58aba106cdfcf37d3 P 007c50364bbf48d9834df4c00fa25f33: Deleting WAL directory at /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786143361779964-25976-0/minicluster-data/ts-0/wal/wals/ac3dba3bcd3e46b58aba106cdfcf37d3
I20260807 22:56:44.440654 763 ts_tablet_manager.cc:1939] T ac3dba3bcd3e46b58aba106cdfcf37d3 P 007c50364bbf48d9834df4c00fa25f33: Deleting consensus metadata
I20260807 22:56:44.440665 764 ts_tablet_manager.cc:1905] T ac3dba3bcd3e46b58aba106cdfcf37d3 P a2aa384d454648d1b5a32985fa30c5b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 22:56:44.443293 32418 catalog_manager.cc:4841] TS 007c50364bbf48d9834df4c00fa25f33 (127.25.94.1:43585): tablet ac3dba3bcd3e46b58aba106cdfcf37d3 (table test-table [id=145dba3947be45b78df83b9778c6b7c1]) successfully deleted
I20260807 22:56:44.447103 764 ts_tablet_manager.cc:1918] T ac3dba3bcd3e46b58aba106cdfcf37d3 P a2aa384d454648d1b5a32985fa30c5b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 22:56:44.447292 764 log.cc:1196] T ac3dba3bcd3e46b58aba106cdfcf37d3 P a2aa384d454648d1b5a32985fa30c5b0: Deleting WAL directory at /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786143361779964-25976-0/minicluster-data/ts-1/wal/wals/ac3dba3bcd3e46b58aba106cdfcf37d3
I20260807 22:56:44.447887 764 ts_tablet_manager.cc:1939] T ac3dba3bcd3e46b58aba106cdfcf37d3 P a2aa384d454648d1b5a32985fa30c5b0: Deleting consensus metadata
I20260807 22:56:44.449930 32423 catalog_manager.cc:4841] TS a2aa384d454648d1b5a32985fa30c5b0 (127.25.94.2:44463): tablet ac3dba3bcd3e46b58aba106cdfcf37d3 (table test-table [id=145dba3947be45b78df83b9778c6b7c1]) successfully deleted
Aug 07 22:56:44 dist-test-slave-z8x9 krb5kdc[32372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143401, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.1@KRBTEST.COM
I20260807 22:56:44.460110 724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:44.426117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36598 (local address 127.25.94.1:43585)
0807 22:56:44.427124 (+ 1007us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:44.427133 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:44.441749 (+ 14616us) server_negotiation.cc:407] Connection header received
0807 22:56:44.441991 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:44.441996 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:44.442097 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:44.442211 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:56:44.443935 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:44.444871 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:44.446839 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:44.447463 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:44.447599 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:44.452301 (+ 4702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:56:44.452320 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:56:44.452351 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:56:44.452405 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:56:44.456404 (+ 3999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:44.457453 (+ 1049us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:44.457464 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:44.457467 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:44.457648 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:44.458152 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:44.458158 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:44.458162 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:44.458650 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:56:44.458993 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:44.459454 (+ 461us) server_negotiation.cc:299] Negotiation successful
0807 22:56:44.459698 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":820}
I20260807 22:56:44.461282 32737 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 22:56:44.586508 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 32554
I20260807 22:56:44.609730 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 32752
Aug 07 22:56:44 dist-test-slave-z8x9 krb5kdc[32372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143401, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.2@KRBTEST.COM
I20260807 22:56:44.627928 727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:44.610674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49922 (local address 127.25.94.2:44463)
0807 22:56:44.610861 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:44.610867 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:44.610885 (+ 18us) server_negotiation.cc:407] Connection header received
0807 22:56:44.611011 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:44.611016 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:44.611088 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:44.611170 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:56:44.612879 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:44.613832 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:44.615818 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:44.616411 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:44.616592 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:44.621794 (+ 5202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:56:44.621809 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:56:44.621828 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:56:44.621863 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:56:44.624770 (+ 2907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:44.625684 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:44.625697 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:44.625700 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:44.625845 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:44.626283 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:44.626288 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:44.626291 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:44.626679 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:56:44.627001 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:44.627309 (+ 308us) server_negotiation.cc:299] Negotiation successful
0807 22:56:44.627556 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260807 22:56:44.628587 484 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 22:56:44.759855 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 32752
I20260807 22:56:44.787410 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 498
Aug 07 22:56:44 dist-test-slave-z8x9 krb5kdc[32372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143401, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.3@KRBTEST.COM
I20260807 22:56:44.804347 728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:44.788299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54370 (local address 127.25.94.3:38231)
0807 22:56:44.788517 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:44.788522 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:44.788544 (+ 22us) server_negotiation.cc:407] Connection header received
0807 22:56:44.788692 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:44.788704 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:44.788774 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:44.788886 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:56:44.790223 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:44.791133 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:44.792415 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:44.792882 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:44.793031 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:44.797716 (+ 4685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:56:44.797729 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:56:44.797745 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:56:44.797784 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:56:44.801108 (+ 3324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:44.802068 (+ 960us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:44.802091 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:44.802094 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:44.802274 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:44.802680 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:44.802686 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:44.802689 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:44.803061 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:56:44.803439 (+ 378us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:44.803727 (+ 288us) server_negotiation.cc:299] Negotiation successful
0807 22:56:44.803966 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260807 22:56:44.805280 691 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 22:56:44.930531 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 498
I20260807 22:56:44.956910 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 32393
I20260807 22:56:44.958292 32488 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 22:56:45.069875 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 32393
2026-08-07T22:56:45Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4840 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-07T22:56:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T22:56:45Z Disabled control of system clock
I20260807 22:56:45.154704 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786143361779964-25976-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786143361779964-25976-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786143361779964-25976-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.62:37389
--webserver_interface=127.25.94.62
--webserver_port=0
--builtin_ntp_servers=127.25.94.20:35531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.94.62:37389
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260807 22:56:45.510082 777 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:56:45.510409 777 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:56:45.510582 777 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:56:45.518064 777 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 22:56:45.518173 777 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:56:45.518267 777 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:56:45.518374 777 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 22:56:45.518457 777 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 22:56:45.528136 777 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:35531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786143361779964-25976-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786143361779964-25976-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.94.62:37389
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.62:37389
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.94.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786143361779964-25976-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 22:56:45.529587 777 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:56:45.531451 777 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:56:45.543705 787 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:56:45.543798 785 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:56:45.543774 790 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:56:45.544679 777 server_base.cc:1034] running on GCE node
I20260807 22:56:45.545608 777 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:56:45.546995 777 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:56:45.548243 777 hybrid_clock.cc:648] HybridClock initialized: now 1786143405548197 us; error 32 us; skew 500 ppm
I20260807 22:56:45.548650 777 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 22:56:45.551165 777 webserver.cc:460] Webserver started at http://127.25.94.62:34979/ using document root <none> and password file <none>
I20260807 22:56:45.551854 777 fs_manager.cc:359] Metadata directory not provided
I20260807 22:56:45.551949 777 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:56:45.552245 777 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:56:45.555522 777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786143361779964-25976-0/minicluster-data/master-0/data/instance:
uuid: "a8f3e2a37acf4a11a7a765223e2b6fba"
format_stamp: "Formatted at 2026-08-07 22:56:45 on dist-test-slave-z8x9"
server_key: "d3fa511f5bc66cfb1abd7c5a1c7b2b21"
server_key_iv: "04fd8d098465b5445651dc85dbc65391"
server_key_version: "encryptionkey@0"
I20260807 22:56:45.556288 777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786143361779964-25976-0/minicluster-data/master-0/wal/instance:
uuid: "a8f3e2a37acf4a11a7a765223e2b6fba"
format_stamp: "Formatted at 2026-08-07 22:56:45 on dist-test-slave-z8x9"
server_key: "d3fa511f5bc66cfb1abd7c5a1c7b2b21"
server_key_iv: "04fd8d098465b5445651dc85dbc65391"
server_key_version: "encryptionkey@0"
I20260807 22:56:45.561623 777 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260807 22:56:45.565100 798 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:56:45.566704 777 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 22:56:45.566912 777 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786143361779964-25976-0/minicluster-data/master-0/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786143361779964-25976-0/minicluster-data/master-0/wal
uuid: "a8f3e2a37acf4a11a7a765223e2b6fba"
format_stamp: "Formatted at 2026-08-07 22:56:45 on dist-test-slave-z8x9"
server_key: "d3fa511f5bc66cfb1abd7c5a1c7b2b21"
server_key_iv: "04fd8d098465b5445651dc85dbc65391"
server_key_version: "encryptionkey@0"
I20260807 22:56:45.567153 777 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786143361779964-25976-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786143361779964-25976-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786143361779964-25976-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:56:45.573544 777 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:56:45.574330 777 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:56:45.574651 777 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:56:45.597355 777 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.62:37389
I20260807 22:56:45.597424 876 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.62:37389 every 8 connection(s)
I20260807 22:56:45.598747 777 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
I20260807 22:56:45.599927 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 777
I20260807 22:56:45.600132 25976 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786143361779964-25976-0/minicluster-data/master-0/wal/instance
I20260807 22:56:45.600778 25976 external_mini_cluster.cc:1428] Setting key f9d07b3571ec46d1309756703651010b
I20260807 22:56:45.603315 877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:56:45.610519 882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:45.602990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56314 (local address 127.25.94.62:37389)
0807 22:56:45.604105 (+ 1115us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:45.604120 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:45.604172 (+ 52us) server_negotiation.cc:407] Connection header received
0807 22:56:45.606354 (+ 2182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:45.606381 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:45.606941 (+ 560us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:45.607387 (+ 446us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:56:45.608439 (+ 1052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:56:45.608447 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:56:45.608462 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 22:56:45.608479 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:56:45.608571 (+ 92us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 22:56:45.608591 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:56:45.608823 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:45.608983 (+ 160us) server_negotiation.cc:299] Negotiation successful
0807 22:56:45.609276 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":588,"thread_start_us":372,"threads_started":1}
I20260807 22:56:45.616499 877 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8f3e2a37acf4a11a7a765223e2b6fba: Bootstrap starting.
I20260807 22:56:45.619688 877 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a8f3e2a37acf4a11a7a765223e2b6fba: Neither blocks nor log segments found. Creating new log.
I20260807 22:56:45.620707 877 log.cc:824] T 00000000000000000000000000000000 P a8f3e2a37acf4a11a7a765223e2b6fba: Log is configured to *not* fsync() on all Append() calls
I20260807 22:56:45.623521 877 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8f3e2a37acf4a11a7a765223e2b6fba: No bootstrap required, opened a new log
I20260807 22:56:45.630730 877 raft_consensus.cc:348] T 00000000000000000000000000000000 P a8f3e2a37acf4a11a7a765223e2b6fba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8f3e2a37acf4a11a7a765223e2b6fba" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 37389 } }
I20260807 22:56:45.631114 877 raft_consensus.cc:374] T 00000000000000000000000000000000 P a8f3e2a37acf4a11a7a765223e2b6fba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:56:45.631232 877 raft_consensus.cc:729] T 00000000000000000000000000000000 P a8f3e2a37acf4a11a7a765223e2b6fba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8f3e2a37acf4a11a7a765223e2b6fba, State: Initialized, Role: FOLLOWER
I20260807 22:56:45.631866 877 consensus_queue.cc:260] T 00000000000000000000000000000000 P a8f3e2a37acf4a11a7a765223e2b6fba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8f3e2a37acf4a11a7a765223e2b6fba" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 37389 } }
I20260807 22:56:45.632113 877 raft_consensus.cc:388] T 00000000000000000000000000000000 P a8f3e2a37acf4a11a7a765223e2b6fba [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 22:56:45.632261 877 raft_consensus.cc:482] T 00000000000000000000000000000000 P a8f3e2a37acf4a11a7a765223e2b6fba [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 22:56:45.632467 877 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a8f3e2a37acf4a11a7a765223e2b6fba [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:56:45.634912 877 raft_consensus.cc:504] T 00000000000000000000000000000000 P a8f3e2a37acf4a11a7a765223e2b6fba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8f3e2a37acf4a11a7a765223e2b6fba" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 37389 } }
I20260807 22:56:45.635483 877 leader_election.cc:302] T 00000000000000000000000000000000 P a8f3e2a37acf4a11a7a765223e2b6fba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a8f3e2a37acf4a11a7a765223e2b6fba; no voters:
I20260807 22:56:45.636229 877 leader_election.cc:288] T 00000000000000000000000000000000 P a8f3e2a37acf4a11a7a765223e2b6fba [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 22:56:45.636375 884 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a8f3e2a37acf4a11a7a765223e2b6fba [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:56:45.637317 884 raft_consensus.cc:686] T 00000000000000000000000000000000 P a8f3e2a37acf4a11a7a765223e2b6fba [term 1 LEADER]: Becoming Leader. State: Replica: a8f3e2a37acf4a11a7a765223e2b6fba, State: Running, Role: LEADER
I20260807 22:56:45.637902 884 consensus_queue.cc:237] T 00000000000000000000000000000000 P a8f3e2a37acf4a11a7a765223e2b6fba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8f3e2a37acf4a11a7a765223e2b6fba" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 37389 } }
I20260807 22:56:45.638594 877 sys_catalog.cc:564] T 00000000000000000000000000000000 P a8f3e2a37acf4a11a7a765223e2b6fba [sys.catalog]: configured and running, proceeding with master startup.
I20260807 22:56:45.641161 886 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8f3e2a37acf4a11a7a765223e2b6fba [sys.catalog]: SysCatalogTable state changed. Reason: New leader a8f3e2a37acf4a11a7a765223e2b6fba. Latest consensus state: current_term: 1 leader_uuid: "a8f3e2a37acf4a11a7a765223e2b6fba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8f3e2a37acf4a11a7a765223e2b6fba" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 37389 } } }
I20260807 22:56:45.641139 885 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8f3e2a37acf4a11a7a765223e2b6fba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a8f3e2a37acf4a11a7a765223e2b6fba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8f3e2a37acf4a11a7a765223e2b6fba" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 37389 } } }
I20260807 22:56:45.641712 885 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8f3e2a37acf4a11a7a765223e2b6fba [sys.catalog]: This master's current role is: LEADER
I20260807 22:56:45.641713 886 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8f3e2a37acf4a11a7a765223e2b6fba [sys.catalog]: This master's current role is: LEADER
I20260807 22:56:45.642814 896 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 22:56:45.649986 896 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 22:56:45.667510 896 catalog_manager.cc:1347] Generated new cluster ID: 9da9e459471346189f7cbec0e080c3f1
I20260807 22:56:45.667652 896 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 22:56:45.727475 896 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 22:56:45.727962 896 catalog_manager.cc:1504] Loading token signing keys...
I20260807 22:56:45.741098 896 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a8f3e2a37acf4a11a7a765223e2b6fba: Generated new TSK 0
I20260807 22:56:45.742260 896 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 22:56:45.761723 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786143361779964-25976-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786143361779964-25976-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786143361779964-25976-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786143361779964-25976-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.1:0
--local_ip_for_outbound_sockets=127.25.94.1
--webserver_interface=127.25.94.1
--webserver_port=0
--tserver_master_addrs=127.25.94.62:37389
--builtin_ntp_servers=127.25.94.20:35531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260807 22:56:46.110816 916 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:56:46.111167 916 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:56:46.111341 916 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:56:46.119402 916 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:56:46.119549 916 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:56:46.119716 916 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.1
I20260807 22:56:46.130033 916 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:35531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786143361779964-25976-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786143361779964-25976-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786143361779964-25976-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.94.1
--webserver_port=0
--tserver_master_addrs=127.25.94.62:37389
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.1
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786143361779964-25976-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 22:56:46.131513 916 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:56:46.133672 916 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:56:46.149559 928 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:56:46.149868 923 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:56:46.149595 926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:56:46.150360 916 server_base.cc:1034] running on GCE node
I20260807 22:56:46.151470 916 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:56:46.152639 916 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:56:46.153963 916 hybrid_clock.cc:648] HybridClock initialized: now 1786143406153870 us; error 86 us; skew 500 ppm
I20260807 22:56:46.154487 916 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 22:56:46.157637 916 webserver.cc:460] Webserver started at http://127.25.94.1:45871/ using document root <none> and password file <none>
I20260807 22:56:46.158483 916 fs_manager.cc:359] Metadata directory not provided
I20260807 22:56:46.158636 916 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:56:46.159014 916 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:56:46.163143 916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786143361779964-25976-0/minicluster-data/ts-0/data/instance:
uuid: "8a32077fbaf941e3b9e055794c468e97"
format_stamp: "Formatted at 2026-08-07 22:56:46 on dist-test-slave-z8x9"
server_key: "576e1b44bb96309cf5788cc8a375c521"
server_key_iv: "4e901af24f2608d65159af514aea05ab"
server_key_version: "encryptionkey@0"
I20260807 22:56:46.164474 916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786143361779964-25976-0/minicluster-data/ts-0/wal/instance:
uuid: "8a32077fbaf941e3b9e055794c468e97"
format_stamp: "Formatted at 2026-08-07 22:56:46 on dist-test-slave-z8x9"
server_key: "576e1b44bb96309cf5788cc8a375c521"
server_key_iv: "4e901af24f2608d65159af514aea05ab"
server_key_version: "encryptionkey@0"
I20260807 22:56:46.173731 916 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.006s sys 0.005s
I20260807 22:56:46.180140 936 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:56:46.183109 916 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.002s sys 0.000s
I20260807 22:56:46.183424 916 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786143361779964-25976-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786143361779964-25976-0/minicluster-data/ts-0/wal
uuid: "8a32077fbaf941e3b9e055794c468e97"
format_stamp: "Formatted at 2026-08-07 22:56:46 on dist-test-slave-z8x9"
server_key: "576e1b44bb96309cf5788cc8a375c521"
server_key_iv: "4e901af24f2608d65159af514aea05ab"
server_key_version: "encryptionkey@0"
I20260807 22:56:46.183789 916 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786143361779964-25976-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786143361779964-25976-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786143361779964-25976-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:56:46.193692 916 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:56:46.196878 916 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:56:46.199064 916 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:56:46.201344 916 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:56:46.203238 916 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:56:46.203384 916 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:56:46.203552 916 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:56:46.203655 916 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:56:46.244745 916 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.1:41073
I20260807 22:56:46.244838 1098 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.1:41073 every 8 connection(s)
I20260807 22:56:46.246318 916 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786143361779964-25976-0/minicluster-data/ts-0/data/info.pb
I20260807 22:56:46.248245 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 916
I20260807 22:56:46.248451 25976 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786143361779964-25976-0/minicluster-data/ts-0/wal/instance
I20260807 22:56:46.249109 25976 external_mini_cluster.cc:1428] Setting key 7d44316e91bc1ab6df52a6e2895fef0b
I20260807 22:56:46.253105 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786143361779964-25976-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786143361779964-25976-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786143361779964-25976-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786143361779964-25976-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.2:0
--local_ip_for_outbound_sockets=127.25.94.2
--webserver_interface=127.25.94.2
--webserver_port=0
--tserver_master_addrs=127.25.94.62:37389
--builtin_ntp_servers=127.25.94.20:35531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260807 22:56:46.257648 1101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:46.251182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.1:53163 (local address 127.25.94.62:37389)
0807 22:56:46.251722 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:46.251728 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:46.253154 (+ 1426us) server_negotiation.cc:407] Connection header received
0807 22:56:46.255073 (+ 1919us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:46.255079 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:46.255167 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:46.255312 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:56:46.256777 (+ 1465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:56:46.256783 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:56:46.256788 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 22:56:46.256807 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:56:46.256861 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 22:56:46.256873 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:56:46.257011 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:46.257273 (+ 262us) server_negotiation.cc:299] Negotiation successful
0807 22:56:46.257347 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":242,"threads_started":1}
I20260807 22:56:46.258723 1102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:46.251919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:37389 (local address 127.25.94.1:53163)
0807 22:56:46.252977 (+ 1058us) negotiation.cc:107] Waiting for socket to connect
0807 22:56:46.253026 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0807 22:56:46.254777 (+ 1751us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:56:46.255577 (+ 800us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:56:46.255586 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:56:46.255685 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:56:46.256462 (+ 777us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 22:56:46.256464 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:56:46.256504 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 22:56:46.256595 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:56:46.256973 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:56:46.256986 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:56:46.256997 (+ 11us) client_negotiation.cc:769] Sending connection context
0807 22:56:46.257361 (+ 364us) client_negotiation.cc:240] Negotiation successful
0807 22:56:46.257639 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":336,"threads_started":1}
I20260807 22:56:46.260324 1099 heartbeater.cc:347] Connected to a master server at 127.25.94.62:37389
I20260807 22:56:46.260670 1099 heartbeater.cc:464] Registering TS with master...
I20260807 22:56:46.261374 1099 heartbeater.cc:511] Master 127.25.94.62:37389 requested a full tablet report, sending...
I20260807 22:56:46.263701 833 ts_manager.cc:194] Registered new tserver with Master: 8a32077fbaf941e3b9e055794c468e97 (127.25.94.1:41073)
W20260807 22:56:46.594470 1103 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:56:46.594954 1103 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:56:46.595116 1103 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:56:46.604015 1103 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:56:46.604151 1103 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:56:46.604423 1103 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.2
I20260807 22:56:46.613966 1103 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:35531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786143361779964-25976-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786143361779964-25976-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786143361779964-25976-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.94.2
--webserver_port=0
--tserver_master_addrs=127.25.94.62:37389
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.2
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786143361779964-25976-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 22:56:46.615273 1103 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:56:46.617290 1103 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:56:46.630542 1112 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:56:46.630499 1116 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:56:46.630574 1119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:56:46.631338 1103 server_base.cc:1034] running on GCE node
I20260807 22:56:46.631969 1103 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:56:46.632988 1103 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:56:46.634220 1103 hybrid_clock.cc:648] HybridClock initialized: now 1786143406634174 us; error 72 us; skew 500 ppm
I20260807 22:56:46.634544 1103 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 22:56:46.636878 1103 webserver.cc:460] Webserver started at http://127.25.94.2:44371/ using document root <none> and password file <none>
I20260807 22:56:46.637650 1103 fs_manager.cc:359] Metadata directory not provided
I20260807 22:56:46.637787 1103 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:56:46.638136 1103 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:56:46.641011 1103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786143361779964-25976-0/minicluster-data/ts-1/data/instance:
uuid: "6ab35919dd3843618803f037405829c7"
format_stamp: "Formatted at 2026-08-07 22:56:46 on dist-test-slave-z8x9"
server_key: "e0b323d61f240b7b24bbf115490253c3"
server_key_iv: "07a67940d1fd6a83a1df09afb04faf3a"
server_key_version: "encryptionkey@0"
I20260807 22:56:46.641901 1103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786143361779964-25976-0/minicluster-data/ts-1/wal/instance:
uuid: "6ab35919dd3843618803f037405829c7"
format_stamp: "Formatted at 2026-08-07 22:56:46 on dist-test-slave-z8x9"
server_key: "e0b323d61f240b7b24bbf115490253c3"
server_key_iv: "07a67940d1fd6a83a1df09afb04faf3a"
server_key_version: "encryptionkey@0"
I20260807 22:56:46.647331 1103 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260807 22:56:46.650797 1130 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:56:46.652065 1103 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260807 22:56:46.652251 1103 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786143361779964-25976-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786143361779964-25976-0/minicluster-data/ts-1/wal
uuid: "6ab35919dd3843618803f037405829c7"
format_stamp: "Formatted at 2026-08-07 22:56:46 on dist-test-slave-z8x9"
server_key: "e0b323d61f240b7b24bbf115490253c3"
server_key_iv: "07a67940d1fd6a83a1df09afb04faf3a"
server_key_version: "encryptionkey@0"
I20260807 22:56:46.652475 1103 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786143361779964-25976-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786143361779964-25976-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786143361779964-25976-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:56:46.658866 1103 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:56:46.659870 1103 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:56:46.660213 1103 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:56:46.661603 1103 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:56:46.665339 1103 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:56:46.665459 1103 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:56:46.665580 1103 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:56:46.665658 1103 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:56:46.700235 1103 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.2:34435
I20260807 22:56:46.700297 1303 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.2:34435 every 8 connection(s)
I20260807 22:56:46.701599 1103 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786143361779964-25976-0/minicluster-data/ts-1/data/info.pb
I20260807 22:56:46.708264 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 1103
I20260807 22:56:46.708501 25976 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786143361779964-25976-0/minicluster-data/ts-1/wal/instance
I20260807 22:56:46.709161 25976 external_mini_cluster.cc:1428] Setting key ca9909fc350e21510e91db3f632879e9
I20260807 22:56:46.710318 1101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:46.704227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.2:54415 (local address 127.25.94.62:37389)
0807 22:56:46.704482 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:46.704488 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:46.705863 (+ 1375us) server_negotiation.cc:407] Connection header received
0807 22:56:46.707530 (+ 1667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:46.707535 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:46.707593 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:46.707686 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:56:46.709444 (+ 1758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:56:46.709449 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:56:46.709452 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 22:56:46.709472 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:56:46.709507 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 22:56:46.709518 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:56:46.709666 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:46.709953 (+ 287us) server_negotiation.cc:299] Negotiation successful
0807 22:56:46.710028 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260807 22:56:46.711548 1306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:46.704716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:37389 (local address 127.25.94.2:54415)
0807 22:56:46.705690 (+ 974us) negotiation.cc:107] Waiting for socket to connect
0807 22:56:46.705737 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0807 22:56:46.707243 (+ 1506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:56:46.707979 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:56:46.707992 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:56:46.708123 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:56:46.708967 (+ 844us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 22:56:46.708971 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:56:46.709027 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 22:56:46.709153 (+ 126us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:56:46.709654 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:56:46.709674 (+ 20us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:56:46.709682 (+ 8us) client_negotiation.cc:769] Sending connection context
0807 22:56:46.710011 (+ 329us) client_negotiation.cc:240] Negotiation successful
0807 22:56:46.710329 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":291,"threads_started":1}
I20260807 22:56:46.712664 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786143361779964-25976-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786143361779964-25976-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786143361779964-25976-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786143361779964-25976-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.3:0
--local_ip_for_outbound_sockets=127.25.94.3
--webserver_interface=127.25.94.3
--webserver_port=0
--tserver_master_addrs=127.25.94.62:37389
--builtin_ntp_servers=127.25.94.20:35531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260807 22:56:46.713264 1304 heartbeater.cc:347] Connected to a master server at 127.25.94.62:37389
I20260807 22:56:46.713582 1304 heartbeater.cc:464] Registering TS with master...
I20260807 22:56:46.714192 1304 heartbeater.cc:511] Master 127.25.94.62:37389 requested a full tablet report, sending...
I20260807 22:56:46.715804 833 ts_manager.cc:194] Registered new tserver with Master: 6ab35919dd3843618803f037405829c7 (127.25.94.2:34435)
W20260807 22:56:47.052910 1308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:56:47.053257 1308 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:56:47.053357 1308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:56:47.060904 1308 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:56:47.061023 1308 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:56:47.061136 1308 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.3
I20260807 22:56:47.070608 1308 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:35531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786143361779964-25976-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786143361779964-25976-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786143361779964-25976-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.94.3
--webserver_port=0
--tserver_master_addrs=127.25.94.62:37389
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.3
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786143361779964-25976-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 22:56:47.072109 1308 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:56:47.074069 1308 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:56:47.087747 1314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:56:47.087811 1318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:56:47.087831 1315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:56:47.088729 1308 server_base.cc:1034] running on GCE node
I20260807 22:56:47.089454 1308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:56:47.090534 1308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:56:47.091785 1308 hybrid_clock.cc:648] HybridClock initialized: now 1786143407091714 us; error 56 us; skew 500 ppm
I20260807 22:56:47.092135 1308 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 22:56:47.094661 1308 webserver.cc:460] Webserver started at http://127.25.94.3:38839/ using document root <none> and password file <none>
I20260807 22:56:47.095410 1308 fs_manager.cc:359] Metadata directory not provided
I20260807 22:56:47.095553 1308 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:56:47.095897 1308 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:56:47.098955 1308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786143361779964-25976-0/minicluster-data/ts-2/data/instance:
uuid: "609ef724addf4940b37c7ba559e89cef"
format_stamp: "Formatted at 2026-08-07 22:56:47 on dist-test-slave-z8x9"
server_key: "10df199057ab4a7005898ad711311802"
server_key_iv: "609ac6cf2afb4f742b3b17a7ddfe937a"
server_key_version: "encryptionkey@0"
I20260807 22:56:47.099772 1308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786143361779964-25976-0/minicluster-data/ts-2/wal/instance:
uuid: "609ef724addf4940b37c7ba559e89cef"
format_stamp: "Formatted at 2026-08-07 22:56:47 on dist-test-slave-z8x9"
server_key: "10df199057ab4a7005898ad711311802"
server_key_iv: "609ac6cf2afb4f742b3b17a7ddfe937a"
server_key_version: "encryptionkey@0"
I20260807 22:56:47.105064 1308 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260807 22:56:47.108881 1327 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:56:47.110404 1308 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260807 22:56:47.110607 1308 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786143361779964-25976-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786143361779964-25976-0/minicluster-data/ts-2/wal
uuid: "609ef724addf4940b37c7ba559e89cef"
format_stamp: "Formatted at 2026-08-07 22:56:47 on dist-test-slave-z8x9"
server_key: "10df199057ab4a7005898ad711311802"
server_key_iv: "609ac6cf2afb4f742b3b17a7ddfe937a"
server_key_version: "encryptionkey@0"
I20260807 22:56:47.110885 1308 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786143361779964-25976-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786143361779964-25976-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786143361779964-25976-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:56:47.116966 1308 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:56:47.117918 1308 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:56:47.118227 1308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:56:47.119247 1308 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:56:47.120816 1308 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:56:47.120950 1308 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:56:47.121083 1308 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:56:47.121248 1308 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:56:47.158646 1308 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.3:43613
I20260807 22:56:47.158706 1490 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.3:43613 every 8 connection(s)
I20260807 22:56:47.160096 1308 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786143361779964-25976-0/minicluster-data/ts-2/data/info.pb
I20260807 22:56:47.166546 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 1308
I20260807 22:56:47.166751 25976 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786143361779964-25976-0/minicluster-data/ts-2/wal/instance
I20260807 22:56:47.167354 25976 external_mini_cluster.cc:1428] Setting key 3af533ba7d81605a2fa3a0fd3b1b3228
I20260807 22:56:47.169726 1101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:47.163074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.3:59771 (local address 127.25.94.62:37389)
0807 22:56:47.163326 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:47.163331 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:47.164858 (+ 1527us) server_negotiation.cc:407] Connection header received
0807 22:56:47.166730 (+ 1872us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:47.166735 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:47.166819 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:47.166952 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:56:47.168623 (+ 1671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:56:47.168629 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:56:47.168637 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 22:56:47.168660 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:56:47.168706 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 22:56:47.168719 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:56:47.168856 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:47.169135 (+ 279us) server_negotiation.cc:299] Negotiation successful
0807 22:56:47.169249 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260807 22:56:47.170888 1494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:47.163597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:37389 (local address 127.25.94.3:59771)
0807 22:56:47.164650 (+ 1053us) negotiation.cc:107] Waiting for socket to connect
0807 22:56:47.164697 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0807 22:56:47.166428 (+ 1731us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:56:47.167263 (+ 835us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:56:47.167275 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:56:47.167450 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:56:47.168295 (+ 845us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 22:56:47.168298 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:56:47.168339 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 22:56:47.168461 (+ 122us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:56:47.168859 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:56:47.168868 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:56:47.168902 (+ 34us) client_negotiation.cc:769] Sending connection context
0807 22:56:47.169282 (+ 380us) client_negotiation.cc:240] Negotiation successful
0807 22:56:47.169605 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":335,"threads_started":1}
I20260807 22:56:47.172379 1491 heartbeater.cc:347] Connected to a master server at 127.25.94.62:37389
I20260807 22:56:47.172714 1491 heartbeater.cc:464] Registering TS with master...
I20260807 22:56:47.173403 1491 heartbeater.cc:511] Master 127.25.94.62:37389 requested a full tablet report, sending...
I20260807 22:56:47.174903 833 ts_manager.cc:194] Registered new tserver with Master: 609ef724addf4940b37c7ba559e89cef (127.25.94.3:43613)
I20260807 22:56:47.184024 25976 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 22:56:47.191679 1101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:47.188889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56326 (local address 127.25.94.62:37389)
0807 22:56:47.189091 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:47.189096 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:47.189744 (+ 648us) server_negotiation.cc:407] Connection header received
0807 22:56:47.189917 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:47.189921 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:47.189977 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:47.190063 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:56:47.191131 (+ 1068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:56:47.191135 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:56:47.191139 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 22:56:47.191154 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:56:47.191187 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 22:56:47.191199 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:56:47.191305 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:47.191415 (+ 110us) server_negotiation.cc:299] Negotiation successful
0807 22:56:47.191465 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260807 22:56:47.196161 833 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56326:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 22:56:47.203089 833 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 22:56:47.236045 1521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:47.231572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51854 (local address 127.25.94.2:34435)
0807 22:56:47.233152 (+ 1580us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:47.233161 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:47.233211 (+ 50us) server_negotiation.cc:407] Connection header received
0807 22:56:47.233290 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:47.233299 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:47.233507 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:47.233681 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:56:47.235264 (+ 1583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:56:47.235272 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:56:47.235286 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 22:56:47.235304 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:56:47.235367 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 22:56:47.235394 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:56:47.235529 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:47.235595 (+ 66us) server_negotiation.cc:299] Negotiation successful
0807 22:56:47.235697 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1425,"thread_start_us":1283,"threads_started":1}
I20260807 22:56:47.236166 1516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:47.216757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.1:41073 (local address 127.0.0.1:46034)
0807 22:56:47.228915 (+ 12158us) negotiation.cc:107] Waiting for socket to connect
0807 22:56:47.228971 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0807 22:56:47.229153 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:56:47.233046 (+ 3893us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:56:47.233055 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:56:47.233148 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:56:47.234191 (+ 1043us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 22:56:47.234197 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:56:47.234242 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 22:56:47.234345 (+ 103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:56:47.235306 (+ 961us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:56:47.235312 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:56:47.235318 (+ 6us) client_negotiation.cc:769] Sending connection context
0807 22:56:47.235484 (+ 166us) client_negotiation.cc:240] Negotiation successful
0807 22:56:47.235612 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11762,"thread_start_us":13908,"threads_started":1}
I20260807 22:56:47.236369 1519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:47.231271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.2:34435 (local address 127.0.0.1:51854)
0807 22:56:47.232544 (+ 1273us) negotiation.cc:107] Waiting for socket to connect
0807 22:56:47.232566 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 22:56:47.232667 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:56:47.234198 (+ 1531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:56:47.234203 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:56:47.234228 (+ 25us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:56:47.235021 (+ 793us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 22:56:47.235024 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:56:47.235041 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 22:56:47.235089 (+ 48us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:56:47.235481 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:56:47.235486 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:56:47.235490 (+ 4us) client_negotiation.cc:769] Sending connection context
0807 22:56:47.235624 (+ 134us) client_negotiation.cc:240] Negotiation successful
0807 22:56:47.235684 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1163,"thread_start_us":1502,"threads_started":1}
I20260807 22:56:47.236080 1518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:47.218285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56178 (local address 127.25.94.3:43613)
0807 22:56:47.230518 (+ 12233us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:47.230526 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:47.231042 (+ 516us) server_negotiation.cc:407] Connection header received
0807 22:56:47.231240 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:47.231247 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:47.231517 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:47.231687 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:56:47.234965 (+ 3278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:56:47.234975 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:56:47.234990 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 22:56:47.235012 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:56:47.235092 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 22:56:47.235116 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:56:47.235316 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:47.235484 (+ 168us) server_negotiation.cc:299] Negotiation successful
0807 22:56:47.235602 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12044,"thread_start_us":11859,"threads_started":1}
I20260807 22:56:47.237082 1512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:47.217602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.3:43613 (local address 127.0.0.1:56178)
0807 22:56:47.230905 (+ 13303us) negotiation.cc:107] Waiting for socket to connect
0807 22:56:47.230920 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 22:56:47.231056 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:56:47.233123 (+ 2067us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:56:47.233127 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:56:47.233148 (+ 21us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:56:47.234665 (+ 1517us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 22:56:47.234671 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:56:47.234690 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 22:56:47.234737 (+ 47us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:56:47.235235 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:56:47.235247 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:56:47.235254 (+ 7us) client_negotiation.cc:769] Sending connection context
0807 22:56:47.236565 (+ 1311us) client_negotiation.cc:240] Negotiation successful
0807 22:56:47.236627 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13179,"thread_start_us":11736,"threads_started":1}
I20260807 22:56:47.237540 1514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:47.216991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46034 (local address 127.25.94.1:41073)
0807 22:56:47.232291 (+ 15300us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:47.232301 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:47.232334 (+ 33us) server_negotiation.cc:407] Connection header received
0807 22:56:47.232425 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:47.232434 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:47.232664 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:47.233733 (+ 1069us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:56:47.235063 (+ 1330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:56:47.235071 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:56:47.235081 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 22:56:47.235106 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:56:47.235153 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 22:56:47.235175 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:56:47.236975 (+ 1800us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:47.237050 (+ 75us) server_negotiation.cc:299] Negotiation successful
0807 22:56:47.237162 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15130,"thread_start_us":14993,"threads_started":1}
I20260807 22:56:47.240729 1207 tablet_service.cc:1467] Processing CreateTablet for tablet d0ffa386f5a349888f458ea2df7990d2 (DEFAULT_TABLE table=test-table [id=5868731280ac432ea8ea1d51581acb6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:56:47.240887 1006 tablet_service.cc:1467] Processing CreateTablet for tablet d0ffa386f5a349888f458ea2df7990d2 (DEFAULT_TABLE table=test-table [id=5868731280ac432ea8ea1d51581acb6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:56:47.240887 1399 tablet_service.cc:1467] Processing CreateTablet for tablet d0ffa386f5a349888f458ea2df7990d2 (DEFAULT_TABLE table=test-table [id=5868731280ac432ea8ea1d51581acb6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:56:47.242367 1207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0ffa386f5a349888f458ea2df7990d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:56:47.242730 1006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0ffa386f5a349888f458ea2df7990d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:56:47.242725 1399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0ffa386f5a349888f458ea2df7990d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:56:47.262130 1522 tablet_bootstrap.cc:492] T d0ffa386f5a349888f458ea2df7990d2 P 8a32077fbaf941e3b9e055794c468e97: Bootstrap starting.
I20260807 22:56:47.263168 1526 tablet_bootstrap.cc:492] T d0ffa386f5a349888f458ea2df7990d2 P 609ef724addf4940b37c7ba559e89cef: Bootstrap starting.
I20260807 22:56:47.263972 1527 tablet_bootstrap.cc:492] T d0ffa386f5a349888f458ea2df7990d2 P 6ab35919dd3843618803f037405829c7: Bootstrap starting.
I20260807 22:56:47.266655 1522 tablet_bootstrap.cc:654] T d0ffa386f5a349888f458ea2df7990d2 P 8a32077fbaf941e3b9e055794c468e97: Neither blocks nor log segments found. Creating new log.
I20260807 22:56:47.267200 1099 heartbeater.cc:503] Master 127.25.94.62:37389 was elected leader, sending a full tablet report...
I20260807 22:56:47.267380 1527 tablet_bootstrap.cc:654] T d0ffa386f5a349888f458ea2df7990d2 P 6ab35919dd3843618803f037405829c7: Neither blocks nor log segments found. Creating new log.
I20260807 22:56:47.268101 1522 log.cc:824] T d0ffa386f5a349888f458ea2df7990d2 P 8a32077fbaf941e3b9e055794c468e97: Log is configured to *not* fsync() on all Append() calls
I20260807 22:56:47.268668 1527 log.cc:824] T d0ffa386f5a349888f458ea2df7990d2 P 6ab35919dd3843618803f037405829c7: Log is configured to *not* fsync() on all Append() calls
I20260807 22:56:47.272090 1522 tablet_bootstrap.cc:492] T d0ffa386f5a349888f458ea2df7990d2 P 8a32077fbaf941e3b9e055794c468e97: No bootstrap required, opened a new log
I20260807 22:56:47.272572 1522 ts_tablet_manager.cc:1397] T d0ffa386f5a349888f458ea2df7990d2 P 8a32077fbaf941e3b9e055794c468e97: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260807 22:56:47.272586 1526 tablet_bootstrap.cc:654] T d0ffa386f5a349888f458ea2df7990d2 P 609ef724addf4940b37c7ba559e89cef: Neither blocks nor log segments found. Creating new log.
I20260807 22:56:47.274024 1526 log.cc:824] T d0ffa386f5a349888f458ea2df7990d2 P 609ef724addf4940b37c7ba559e89cef: Log is configured to *not* fsync() on all Append() calls
I20260807 22:56:47.276151 1527 tablet_bootstrap.cc:492] T d0ffa386f5a349888f458ea2df7990d2 P 6ab35919dd3843618803f037405829c7: No bootstrap required, opened a new log
I20260807 22:56:47.276467 1527 ts_tablet_manager.cc:1397] T d0ffa386f5a349888f458ea2df7990d2 P 6ab35919dd3843618803f037405829c7: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.007s
I20260807 22:56:47.277305 1526 tablet_bootstrap.cc:492] T d0ffa386f5a349888f458ea2df7990d2 P 609ef724addf4940b37c7ba559e89cef: No bootstrap required, opened a new log
I20260807 22:56:47.277634 1526 ts_tablet_manager.cc:1397] T d0ffa386f5a349888f458ea2df7990d2 P 609ef724addf4940b37c7ba559e89cef: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.006s
I20260807 22:56:47.280613 1522 raft_consensus.cc:348] T d0ffa386f5a349888f458ea2df7990d2 P 8a32077fbaf941e3b9e055794c468e97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a32077fbaf941e3b9e055794c468e97" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 41073 } } peers { permanent_uuid: "6ab35919dd3843618803f037405829c7" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 34435 } } peers { permanent_uuid: "609ef724addf4940b37c7ba559e89cef" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 43613 } }
I20260807 22:56:47.280992 1522 raft_consensus.cc:374] T d0ffa386f5a349888f458ea2df7990d2 P 8a32077fbaf941e3b9e055794c468e97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:56:47.281106 1522 raft_consensus.cc:729] T d0ffa386f5a349888f458ea2df7990d2 P 8a32077fbaf941e3b9e055794c468e97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a32077fbaf941e3b9e055794c468e97, State: Initialized, Role: FOLLOWER
I20260807 22:56:47.281841 1522 consensus_queue.cc:260] T d0ffa386f5a349888f458ea2df7990d2 P 8a32077fbaf941e3b9e055794c468e97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a32077fbaf941e3b9e055794c468e97" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 41073 } } peers { permanent_uuid: "6ab35919dd3843618803f037405829c7" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 34435 } } peers { permanent_uuid: "609ef724addf4940b37c7ba559e89cef" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 43613 } }
I20260807 22:56:47.283671 1522 ts_tablet_manager.cc:1428] T d0ffa386f5a349888f458ea2df7990d2 P 8a32077fbaf941e3b9e055794c468e97: Time spent starting tablet: real 0.011s user 0.001s sys 0.009s
I20260807 22:56:47.288342 1527 raft_consensus.cc:348] T d0ffa386f5a349888f458ea2df7990d2 P 6ab35919dd3843618803f037405829c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a32077fbaf941e3b9e055794c468e97" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 41073 } } peers { permanent_uuid: "6ab35919dd3843618803f037405829c7" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 34435 } } peers { permanent_uuid: "609ef724addf4940b37c7ba559e89cef" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 43613 } }
I20260807 22:56:47.288823 1527 raft_consensus.cc:374] T d0ffa386f5a349888f458ea2df7990d2 P 6ab35919dd3843618803f037405829c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:56:47.288954 1527 raft_consensus.cc:729] T d0ffa386f5a349888f458ea2df7990d2 P 6ab35919dd3843618803f037405829c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ab35919dd3843618803f037405829c7, State: Initialized, Role: FOLLOWER
I20260807 22:56:47.289259 1526 raft_consensus.cc:348] T d0ffa386f5a349888f458ea2df7990d2 P 609ef724addf4940b37c7ba559e89cef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a32077fbaf941e3b9e055794c468e97" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 41073 } } peers { permanent_uuid: "6ab35919dd3843618803f037405829c7" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 34435 } } peers { permanent_uuid: "609ef724addf4940b37c7ba559e89cef" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 43613 } }
I20260807 22:56:47.289767 1526 raft_consensus.cc:374] T d0ffa386f5a349888f458ea2df7990d2 P 609ef724addf4940b37c7ba559e89cef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:56:47.289880 1526 raft_consensus.cc:729] T d0ffa386f5a349888f458ea2df7990d2 P 609ef724addf4940b37c7ba559e89cef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 609ef724addf4940b37c7ba559e89cef, State: Initialized, Role: FOLLOWER
I20260807 22:56:47.289783 1527 consensus_queue.cc:260] T d0ffa386f5a349888f458ea2df7990d2 P 6ab35919dd3843618803f037405829c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a32077fbaf941e3b9e055794c468e97" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 41073 } } peers { permanent_uuid: "6ab35919dd3843618803f037405829c7" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 34435 } } peers { permanent_uuid: "609ef724addf4940b37c7ba559e89cef" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 43613 } }
I20260807 22:56:47.290686 1526 consensus_queue.cc:260] T d0ffa386f5a349888f458ea2df7990d2 P 609ef724addf4940b37c7ba559e89cef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a32077fbaf941e3b9e055794c468e97" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 41073 } } peers { permanent_uuid: "6ab35919dd3843618803f037405829c7" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 34435 } } peers { permanent_uuid: "609ef724addf4940b37c7ba559e89cef" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 43613 } }
I20260807 22:56:47.291108 1304 heartbeater.cc:503] Master 127.25.94.62:37389 was elected leader, sending a full tablet report...
I20260807 22:56:47.291792 1491 heartbeater.cc:503] Master 127.25.94.62:37389 was elected leader, sending a full tablet report...
I20260807 22:56:47.291826 1527 ts_tablet_manager.cc:1428] T d0ffa386f5a349888f458ea2df7990d2 P 6ab35919dd3843618803f037405829c7: Time spent starting tablet: real 0.015s user 0.015s sys 0.001s
I20260807 22:56:47.292318 1526 ts_tablet_manager.cc:1428] T d0ffa386f5a349888f458ea2df7990d2 P 609ef724addf4940b37c7ba559e89cef: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
W20260807 22:56:47.411990 1493 tablet.cc:2369] T d0ffa386f5a349888f458ea2df7990d2 P 609ef724addf4940b37c7ba559e89cef: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 22:56:47.424319 1535 raft_consensus.cc:482] T d0ffa386f5a349888f458ea2df7990d2 P 6ab35919dd3843618803f037405829c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 22:56:47.424661 1535 raft_consensus.cc:504] T d0ffa386f5a349888f458ea2df7990d2 P 6ab35919dd3843618803f037405829c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a32077fbaf941e3b9e055794c468e97" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 41073 } } peers { permanent_uuid: "6ab35919dd3843618803f037405829c7" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 34435 } } peers { permanent_uuid: "609ef724addf4940b37c7ba559e89cef" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 43613 } }
I20260807 22:56:47.426395 1535 leader_election.cc:288] T d0ffa386f5a349888f458ea2df7990d2 P 6ab35919dd3843618803f037405829c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a32077fbaf941e3b9e055794c468e97 (127.25.94.1:41073), 609ef724addf4940b37c7ba559e89cef (127.25.94.3:43613)
I20260807 22:56:47.429967 1518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:47.426947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.2:39989 (local address 127.25.94.3:43613)
0807 22:56:47.427133 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:47.427140 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:47.427988 (+ 848us) server_negotiation.cc:407] Connection header received
0807 22:56:47.428113 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:47.428118 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:47.428172 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:47.428258 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:56:47.429321 (+ 1063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:56:47.429327 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:56:47.429331 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 22:56:47.429351 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:56:47.429391 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 22:56:47.429408 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:56:47.429539 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:47.429628 (+ 89us) server_negotiation.cc:299] Negotiation successful
0807 22:56:47.429690 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260807 22:56:47.429970 1514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:47.426953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.2:44573 (local address 127.25.94.1:41073)
0807 22:56:47.427134 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:47.427141 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:47.427936 (+ 795us) server_negotiation.cc:407] Connection header received
0807 22:56:47.428402 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:47.428406 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:47.428453 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:47.428524 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:56:47.429339 (+ 815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:56:47.429345 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:56:47.429349 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 22:56:47.429366 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:56:47.429399 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 22:56:47.429413 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:56:47.429532 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:47.429628 (+ 96us) server_negotiation.cc:299] Negotiation successful
0807 22:56:47.429710 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260807 22:56:47.430286 1537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:47.426734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.3:43613 (local address 127.25.94.2:39989)
0807 22:56:47.427801 (+ 1067us) negotiation.cc:107] Waiting for socket to connect
0807 22:56:47.427828 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0807 22:56:47.427985 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:56:47.428307 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:56:47.428310 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:56:47.428329 (+ 19us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:56:47.429029 (+ 700us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 22:56:47.429034 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:56:47.429052 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 22:56:47.429123 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:56:47.429518 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:56:47.429523 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:56:47.429528 (+ 5us) client_negotiation.cc:769] Sending connection context
0807 22:56:47.429734 (+ 206us) client_negotiation.cc:240] Negotiation successful
0807 22:56:47.429797 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":892,"thread_start_us":602,"threads_started":1}
I20260807 22:56:47.430727 1539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:47.426849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.1:41073 (local address 127.25.94.2:44573)
0807 22:56:47.427828 (+ 979us) negotiation.cc:107] Waiting for socket to connect
0807 22:56:47.427843 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 22:56:47.428004 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:56:47.428600 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:56:47.428605 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:56:47.428631 (+ 26us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:56:47.429133 (+ 502us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 22:56:47.429135 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:56:47.429146 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 22:56:47.429213 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:56:47.429518 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:56:47.429523 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:56:47.429528 (+ 5us) client_negotiation.cc:769] Sending connection context
0807 22:56:47.429723 (+ 195us) client_negotiation.cc:240] Negotiation successful
0807 22:56:47.429809 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":876,"mutex_wait_us":26,"spinlock_wait_cycles":49024,"thread_start_us":569,"threads_started":1}
I20260807 22:56:47.431373 1428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0ffa386f5a349888f458ea2df7990d2" candidate_uuid: "6ab35919dd3843618803f037405829c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "609ef724addf4940b37c7ba559e89cef" is_pre_election: true
I20260807 22:56:47.431869 1428 raft_consensus.cc:2454] T d0ffa386f5a349888f458ea2df7990d2 P 609ef724addf4940b37c7ba559e89cef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ab35919dd3843618803f037405829c7 in term 0.
I20260807 22:56:47.431907 1036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0ffa386f5a349888f458ea2df7990d2" candidate_uuid: "6ab35919dd3843618803f037405829c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a32077fbaf941e3b9e055794c468e97" is_pre_election: true
I20260807 22:56:47.432427 1036 raft_consensus.cc:2454] T d0ffa386f5a349888f458ea2df7990d2 P 8a32077fbaf941e3b9e055794c468e97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ab35919dd3843618803f037405829c7 in term 0.
I20260807 22:56:47.432539 1131 leader_election.cc:302] T d0ffa386f5a349888f458ea2df7990d2 P 6ab35919dd3843618803f037405829c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 609ef724addf4940b37c7ba559e89cef, 6ab35919dd3843618803f037405829c7; no voters:
I20260807 22:56:47.432981 1535 raft_consensus.cc:2781] T d0ffa386f5a349888f458ea2df7990d2 P 6ab35919dd3843618803f037405829c7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 22:56:47.433143 1535 raft_consensus.cc:482] T d0ffa386f5a349888f458ea2df7990d2 P 6ab35919dd3843618803f037405829c7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 22:56:47.433279 1535 raft_consensus.cc:3037] T d0ffa386f5a349888f458ea2df7990d2 P 6ab35919dd3843618803f037405829c7 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:56:47.435480 1535 raft_consensus.cc:504] T d0ffa386f5a349888f458ea2df7990d2 P 6ab35919dd3843618803f037405829c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a32077fbaf941e3b9e055794c468e97" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 41073 } } peers { permanent_uuid: "6ab35919dd3843618803f037405829c7" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 34435 } } peers { permanent_uuid: "609ef724addf4940b37c7ba559e89cef" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 43613 } }
I20260807 22:56:47.436100 1535 leader_election.cc:288] T d0ffa386f5a349888f458ea2df7990d2 P 6ab35919dd3843618803f037405829c7 [CANDIDATE]: Term 1 election: Requested vote from peers 8a32077fbaf941e3b9e055794c468e97 (127.25.94.1:41073), 609ef724addf4940b37c7ba559e89cef (127.25.94.3:43613)
I20260807 22:56:47.436535 1036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0ffa386f5a349888f458ea2df7990d2" candidate_uuid: "6ab35919dd3843618803f037405829c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a32077fbaf941e3b9e055794c468e97"
I20260807 22:56:47.436692 1428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0ffa386f5a349888f458ea2df7990d2" candidate_uuid: "6ab35919dd3843618803f037405829c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "609ef724addf4940b37c7ba559e89cef"
I20260807 22:56:47.436750 1036 raft_consensus.cc:3037] T d0ffa386f5a349888f458ea2df7990d2 P 8a32077fbaf941e3b9e055794c468e97 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:56:47.436928 1428 raft_consensus.cc:3037] T d0ffa386f5a349888f458ea2df7990d2 P 609ef724addf4940b37c7ba559e89cef [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:56:47.439200 1036 raft_consensus.cc:2454] T d0ffa386f5a349888f458ea2df7990d2 P 8a32077fbaf941e3b9e055794c468e97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ab35919dd3843618803f037405829c7 in term 1.
I20260807 22:56:47.439478 1428 raft_consensus.cc:2454] T d0ffa386f5a349888f458ea2df7990d2 P 609ef724addf4940b37c7ba559e89cef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ab35919dd3843618803f037405829c7 in term 1.
I20260807 22:56:47.439652 1132 leader_election.cc:302] T d0ffa386f5a349888f458ea2df7990d2 P 6ab35919dd3843618803f037405829c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ab35919dd3843618803f037405829c7, 8a32077fbaf941e3b9e055794c468e97; no voters:
I20260807 22:56:47.439993 1535 raft_consensus.cc:2781] T d0ffa386f5a349888f458ea2df7990d2 P 6ab35919dd3843618803f037405829c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:56:47.440480 1535 raft_consensus.cc:686] T d0ffa386f5a349888f458ea2df7990d2 P 6ab35919dd3843618803f037405829c7 [term 1 LEADER]: Becoming Leader. State: Replica: 6ab35919dd3843618803f037405829c7, State: Running, Role: LEADER
I20260807 22:56:47.440949 1535 consensus_queue.cc:237] T d0ffa386f5a349888f458ea2df7990d2 P 6ab35919dd3843618803f037405829c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a32077fbaf941e3b9e055794c468e97" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 41073 } } peers { permanent_uuid: "6ab35919dd3843618803f037405829c7" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 34435 } } peers { permanent_uuid: "609ef724addf4940b37c7ba559e89cef" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 43613 } }
I20260807 22:56:47.445619 833 catalog_manager.cc:5495] T d0ffa386f5a349888f458ea2df7990d2 P 6ab35919dd3843618803f037405829c7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6ab35919dd3843618803f037405829c7 (127.25.94.2). New cstate: current_term: 1 leader_uuid: "6ab35919dd3843618803f037405829c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a32077fbaf941e3b9e055794c468e97" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 41073 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ab35919dd3843618803f037405829c7" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 34435 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "609ef724addf4940b37c7ba559e89cef" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 43613 } health_report { overall_health: UNKNOWN } } }
W20260807 22:56:47.453837 1305 tablet.cc:2369] T d0ffa386f5a349888f458ea2df7990d2 P 6ab35919dd3843618803f037405829c7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 22:56:47.466315 1521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:47.464064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51856 (local address 127.25.94.2:34435)
0807 22:56:47.464246 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:47.464251 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:47.464271 (+ 20us) server_negotiation.cc:407] Connection header received
0807 22:56:47.464552 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:47.464557 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:47.464645 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:47.464780 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:56:47.465769 (+ 989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:56:47.465774 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:56:47.465777 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 22:56:47.465794 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:56:47.465829 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 22:56:47.465841 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:56:47.465959 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:47.466027 (+ 68us) server_negotiation.cc:299] Negotiation successful
0807 22:56:47.466075 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260807 22:56:47.475054 1036 raft_consensus.cc:1261] T d0ffa386f5a349888f458ea2df7990d2 P 8a32077fbaf941e3b9e055794c468e97 [term 1 FOLLOWER]: Refusing update from remote peer 6ab35919dd3843618803f037405829c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 22:56:47.475286 1428 raft_consensus.cc:1261] T d0ffa386f5a349888f458ea2df7990d2 P 609ef724addf4940b37c7ba559e89cef [term 1 FOLLOWER]: Refusing update from remote peer 6ab35919dd3843618803f037405829c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 22:56:47.476090 1541 consensus_queue.cc:1035] T d0ffa386f5a349888f458ea2df7990d2 P 6ab35919dd3843618803f037405829c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a32077fbaf941e3b9e055794c468e97" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 41073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 22:56:47.476449 1535 consensus_queue.cc:1035] T d0ffa386f5a349888f458ea2df7990d2 P 6ab35919dd3843618803f037405829c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "609ef724addf4940b37c7ba559e89cef" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 43613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 22:56:47.489099 1551 mvcc.cc:205] Tried to move back new op lower bound from 7316043397007253504 to 7316043396882112512. Current Snapshot: MvccSnapshot[applied={T|T < 7316043397007253504}]
I20260807 22:56:47.489504 1547 mvcc.cc:205] Tried to move back new op lower bound from 7316043397007253504 to 7316043396882112512. Current Snapshot: MvccSnapshot[applied={T|T < 7316043397007253504}]
I20260807 22:56:47.506554 833 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56326:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 22:56:47.506868 833 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56326:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 22:56:47.510915 833 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a8f3e2a37acf4a11a7a765223e2b6fba: Sending DeleteTablet for 3 replicas of tablet d0ffa386f5a349888f458ea2df7990d2
I20260807 22:56:47.512283 1399 tablet_service.cc:1514] Processing DeleteTablet for tablet d0ffa386f5a349888f458ea2df7990d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:56:47 UTC) from {username='slave'} at 127.0.0.1:56178
I20260807 22:56:47.512346 1207 tablet_service.cc:1514] Processing DeleteTablet for tablet d0ffa386f5a349888f458ea2df7990d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:56:47 UTC) from {username='slave'} at 127.0.0.1:51854
I20260807 22:56:47.512612 1006 tablet_service.cc:1514] Processing DeleteTablet for tablet d0ffa386f5a349888f458ea2df7990d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:56:47 UTC) from {username='slave'} at 127.0.0.1:46034
I20260807 22:56:47.516975 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 916
I20260807 22:56:47.527915 1570 tablet_replica.cc:330] T d0ffa386f5a349888f458ea2df7990d2 P 6ab35919dd3843618803f037405829c7: stopping tablet replica
I20260807 22:56:47.528775 1570 raft_consensus.cc:2229] T d0ffa386f5a349888f458ea2df7990d2 P 6ab35919dd3843618803f037405829c7 [term 1 LEADER]: Raft consensus shutting down.
I20260807 22:56:47.529556 1570 raft_consensus.cc:2258] T d0ffa386f5a349888f458ea2df7990d2 P 6ab35919dd3843618803f037405829c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 22:56:47.531857 1570 ts_tablet_manager.cc:1905] T d0ffa386f5a349888f458ea2df7990d2 P 6ab35919dd3843618803f037405829c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 22:56:47.537698 1514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:47.518242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46048 (local address 127.25.94.1:41073)
0807 22:56:47.518424 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:47.518431 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:47.534121 (+ 15690us) server_negotiation.cc:407] Connection header received
0807 22:56:47.534754 (+ 633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:47.534762 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:47.534918 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:47.535016 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:56:47.536563 (+ 1547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:56:47.536573 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:56:47.536581 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 22:56:47.536611 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:56:47.536663 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 22:56:47.536684 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:56:47.536814 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:47.537103 (+ 289us) server_negotiation.cc:299] Negotiation successful
0807 22:56:47.537240 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260807 22:56:47.539188 1089 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 22:56:47.539777 1570 ts_tablet_manager.cc:1918] T d0ffa386f5a349888f458ea2df7990d2 P 6ab35919dd3843618803f037405829c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 22:56:47.539965 1570 log.cc:1196] T d0ffa386f5a349888f458ea2df7990d2 P 6ab35919dd3843618803f037405829c7: Deleting WAL directory at /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786143361779964-25976-0/minicluster-data/ts-1/wal/wals/d0ffa386f5a349888f458ea2df7990d2
I20260807 22:56:47.540548 1567 tablet_replica.cc:330] T d0ffa386f5a349888f458ea2df7990d2 P 609ef724addf4940b37c7ba559e89cef: stopping tablet replica
I20260807 22:56:47.540601 1570 ts_tablet_manager.cc:1939] T d0ffa386f5a349888f458ea2df7990d2 P 6ab35919dd3843618803f037405829c7: Deleting consensus metadata
I20260807 22:56:47.541260 1567 raft_consensus.cc:2229] T d0ffa386f5a349888f458ea2df7990d2 P 609ef724addf4940b37c7ba559e89cef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 22:56:47.541972 1567 raft_consensus.cc:2258] T d0ffa386f5a349888f458ea2df7990d2 P 609ef724addf4940b37c7ba559e89cef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 22:56:47.542667 802 catalog_manager.cc:4841] TS 6ab35919dd3843618803f037405829c7 (127.25.94.2:34435): tablet d0ffa386f5a349888f458ea2df7990d2 (table test-table [id=5868731280ac432ea8ea1d51581acb6b]) successfully deleted
I20260807 22:56:47.544893 1567 ts_tablet_manager.cc:1905] T d0ffa386f5a349888f458ea2df7990d2 P 609ef724addf4940b37c7ba559e89cef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 22:56:47.551151 1567 ts_tablet_manager.cc:1918] T d0ffa386f5a349888f458ea2df7990d2 P 609ef724addf4940b37c7ba559e89cef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 22:56:47.551347 1567 log.cc:1196] T d0ffa386f5a349888f458ea2df7990d2 P 609ef724addf4940b37c7ba559e89cef: Deleting WAL directory at /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786143361779964-25976-0/minicluster-data/ts-2/wal/wals/d0ffa386f5a349888f458ea2df7990d2
I20260807 22:56:47.551808 1567 ts_tablet_manager.cc:1939] T d0ffa386f5a349888f458ea2df7990d2 P 609ef724addf4940b37c7ba559e89cef: Deleting consensus metadata
I20260807 22:56:47.553608 799 catalog_manager.cc:4841] TS 609ef724addf4940b37c7ba559e89cef (127.25.94.3:43613): tablet d0ffa386f5a349888f458ea2df7990d2 (table test-table [id=5868731280ac432ea8ea1d51581acb6b]) successfully deleted
I20260807 22:56:47.684197 1571 tablet_replica.cc:330] T d0ffa386f5a349888f458ea2df7990d2 P 8a32077fbaf941e3b9e055794c468e97: stopping tablet replica
I20260807 22:56:47.684760 1571 raft_consensus.cc:2229] T d0ffa386f5a349888f458ea2df7990d2 P 8a32077fbaf941e3b9e055794c468e97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 22:56:47.685134 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 916
W20260807 22:56:47.716570 802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.25.94.1:41073 (error 108)
I20260807 22:56:47.717052 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 1103
I20260807 22:56:47.719874 1512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:47.719303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.1:41073 (local address 127.0.0.1:46060)
0807 22:56:47.719495 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0807 22:56:47.719712 (+ 217us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.1:41073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260807 22:56:47.720919 1521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:47.718407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51858 (local address 127.25.94.2:34435)
0807 22:56:47.718590 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:47.718596 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:47.718615 (+ 19us) server_negotiation.cc:407] Connection header received
0807 22:56:47.718710 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:47.718716 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:47.718783 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:47.718901 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:56:47.719983 (+ 1082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:56:47.719989 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:56:47.719995 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 22:56:47.720019 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:56:47.720063 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 22:56:47.720079 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:56:47.720292 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:47.720544 (+ 252us) server_negotiation.cc:299] Negotiation successful
0807 22:56:47.720606 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260807 22:56:47.720927 802 catalog_manager.cc:4584] TS 8a32077fbaf941e3b9e055794c468e97 (127.25.94.1:41073): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d0ffa386f5a349888f458ea2df7990d2: Network error: Client connection negotiation failed: client connection to 127.25.94.1:41073: connect: Connection refused (error 111)
I20260807 22:56:47.722216 1297 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 22:56:47.774835 1512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:47.774350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.1:41073 (local address 127.0.0.1:46076)
0807 22:56:47.774567 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0807 22:56:47.774661 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.1:41073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260807 22:56:47.775853 1512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:47.775506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.1:41073 (local address 127.0.0.1:46086)
0807 22:56:47.775651 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0807 22:56:47.775745 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.1:41073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260807 22:56:47.835242 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 1103
I20260807 22:56:47.839425 1512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:47.838828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.1:41073 (local address 127.0.0.1:46088)
0807 22:56:47.839031 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0807 22:56:47.839154 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.1:41073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260807 22:56:47.840695 1512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:47.840286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.1:41073 (local address 127.0.0.1:46090)
0807 22:56:47.840459 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0807 22:56:47.840547 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.1:41073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260807 22:56:47.863909 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 1308
I20260807 22:56:47.871260 1518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:47.864742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56184 (local address 127.25.94.3:43613)
0807 22:56:47.864977 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:47.864982 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:47.865002 (+ 20us) server_negotiation.cc:407] Connection header received
0807 22:56:47.865083 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:47.865087 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:47.865264 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:47.865359 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:56:47.870153 (+ 4794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:56:47.870159 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:56:47.870163 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 22:56:47.870185 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:56:47.870223 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 22:56:47.870238 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:56:47.870369 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:47.870777 (+ 408us) server_negotiation.cc:299] Negotiation successful
0807 22:56:47.870994 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260807 22:56:47.872058 1486 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 22:56:47.921393 1512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:47.920738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.1:41073 (local address 127.0.0.1:46094)
0807 22:56:47.920992 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0807 22:56:47.921125 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.1:41073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260807 22:56:47.922475 1512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:47.922069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.1:41073 (local address 127.0.0.1:46108)
0807 22:56:47.922269 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0807 22:56:47.922352 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.1:41073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260807 22:56:47.962404 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 1308
I20260807 22:56:47.988487 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 777
I20260807 22:56:47.989967 871 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 22:56:48.096769 1512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:48.096053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.1:41073 (local address 127.0.0.1:46122)
0807 22:56:48.096268 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0807 22:56:48.096379 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.1:41073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260807 22:56:48.096890 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 777
2026-08-07T22:56:48Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (3028 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260807 22:56:48.152148 25976 mini_oidc.cc:152] Starting JWKS server
I20260807 22:56:48.155709 25976 webserver.cc:460] Webserver started at https://127.0.0.1:35209/ using document root <none> and password file <none>
I20260807 22:56:48.156164 25976 mini_oidc.cc:189] Starting OIDC Discovery server
I20260807 22:56:48.157275 25976 webserver.cc:460] Webserver started at http://127.0.0.1:38611/ using document root <none> and password file <none>
2026-08-07T22:56:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T22:56:48Z Disabled control of system clock
I20260807 22:56:48.194871 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786143361779964-25976-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786143361779964-25976-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786143361779964-25976-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.62:35857
--webserver_interface=127.25.94.62
--webserver_port=0
--builtin_ntp_servers=127.25.94.20:34655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.94.62:35857
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35209/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786143361779964-25976-0/testchainca.cert with env {}
W20260807 22:56:48.543457 1587 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:56:48.543830 1587 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:56:48.544003 1587 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:56:48.551311 1587 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 22:56:48.551492 1587 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:56:48.551589 1587 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:56:48.551697 1587 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 22:56:48.551789 1587 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260807 22:56:48.551882 1587 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260807 22:56:48.551965 1587 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:35209/jwks.json
I20260807 22:56:48.562847 1587 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:34655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786143361779964-25976-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786143361779964-25976-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.94.62:35857
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.62:35857
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35209/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.94.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786143361779964-25976-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786143361779964-25976-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 22:56:48.564484 1587 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:56:48.565910 1587 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:56:48.578400 1597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:56:48.578440 1596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:56:48.578351 1600 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:56:48.582930 1587 server_base.cc:1034] running on GCE node
I20260807 22:56:48.583858 1587 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:56:48.585502 1587 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:56:48.586758 1587 hybrid_clock.cc:648] HybridClock initialized: now 1786143408586711 us; error 46 us; skew 500 ppm
I20260807 22:56:48.587126 1587 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 22:56:48.590121 1587 webserver.cc:460] Webserver started at http://127.25.94.62:44799/ using document root <none> and password file <none>
I20260807 22:56:48.590875 1587 fs_manager.cc:359] Metadata directory not provided
I20260807 22:56:48.591025 1587 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:56:48.591375 1587 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:56:48.594583 1587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786143361779964-25976-0/minicluster-data/master-0/data/instance:
uuid: "5b7d551137fd46e0b3f53a5144d3bf4e"
format_stamp: "Formatted at 2026-08-07 22:56:48 on dist-test-slave-z8x9"
server_key: "8e79fb3dfa5e1b2a0dddf90beadf56a9"
server_key_iv: "03f1fe0f7073065f046b4d1e5d3e3fb3"
server_key_version: "encryptionkey@0"
I20260807 22:56:48.595446 1587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786143361779964-25976-0/minicluster-data/master-0/wal/instance:
uuid: "5b7d551137fd46e0b3f53a5144d3bf4e"
format_stamp: "Formatted at 2026-08-07 22:56:48 on dist-test-slave-z8x9"
server_key: "8e79fb3dfa5e1b2a0dddf90beadf56a9"
server_key_iv: "03f1fe0f7073065f046b4d1e5d3e3fb3"
server_key_version: "encryptionkey@0"
I20260807 22:56:48.601576 1587 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260807 22:56:48.606185 1608 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:56:48.607611 1587 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 22:56:48.607774 1587 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786143361779964-25976-0/minicluster-data/master-0/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786143361779964-25976-0/minicluster-data/master-0/wal
uuid: "5b7d551137fd46e0b3f53a5144d3bf4e"
format_stamp: "Formatted at 2026-08-07 22:56:48 on dist-test-slave-z8x9"
server_key: "8e79fb3dfa5e1b2a0dddf90beadf56a9"
server_key_iv: "03f1fe0f7073065f046b4d1e5d3e3fb3"
server_key_version: "encryptionkey@0"
I20260807 22:56:48.607985 1587 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786143361779964-25976-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786143361779964-25976-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786143361779964-25976-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:56:48.629724 1587 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:56:48.630560 1587 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:56:48.630852 1587 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:56:48.650414 1587 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.62:35857
I20260807 22:56:48.650434 1689 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.62:35857 every 8 connection(s)
I20260807 22:56:48.651790 1587 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
I20260807 22:56:48.655903 1690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:56:48.661396 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 1587
I20260807 22:56:48.661618 25976 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786143361779964-25976-0/minicluster-data/master-0/wal/instance
I20260807 22:56:48.662436 25976 external_mini_cluster.cc:1428] Setting key a453d117d074310027f7d321c0f57c83
I20260807 22:56:48.669569 1690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b7d551137fd46e0b3f53a5144d3bf4e: Bootstrap starting.
I20260807 22:56:48.674371 1690 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5b7d551137fd46e0b3f53a5144d3bf4e: Neither blocks nor log segments found. Creating new log.
I20260807 22:56:48.675882 1690 log.cc:824] T 00000000000000000000000000000000 P 5b7d551137fd46e0b3f53a5144d3bf4e: Log is configured to *not* fsync() on all Append() calls
I20260807 22:56:48.680866 1690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b7d551137fd46e0b3f53a5144d3bf4e: No bootstrap required, opened a new log
I20260807 22:56:48.689819 1694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:48.670579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41144 (local address 127.25.94.62:35857)
0807 22:56:48.674786 (+ 4207us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:48.674813 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:48.674882 (+ 69us) server_negotiation.cc:407] Connection header received
0807 22:56:48.677967 (+ 3085us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:48.678001 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:48.678597 (+ 596us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:48.679892 (+ 1295us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:56:48.680461 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:48.682451 (+ 1990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:48.684233 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:48.685326 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:48.685529 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:48.686755 (+ 1226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:56:48.686766 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:56:48.686786 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 22:56:48.687600 (+ 814us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:56:48.687687 (+ 87us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 22:56:48.687714 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:56:48.687828 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:48.688046 (+ 218us) server_negotiation.cc:299] Negotiation successful
0807 22:56:48.688431 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3555,"thread_start_us":3309,"threads_started":1}
I20260807 22:56:48.690802 1690 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5b7d551137fd46e0b3f53a5144d3bf4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b7d551137fd46e0b3f53a5144d3bf4e" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 35857 } }
I20260807 22:56:48.691260 1690 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5b7d551137fd46e0b3f53a5144d3bf4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:56:48.691397 1690 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5b7d551137fd46e0b3f53a5144d3bf4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b7d551137fd46e0b3f53a5144d3bf4e, State: Initialized, Role: FOLLOWER
I20260807 22:56:48.692055 1690 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5b7d551137fd46e0b3f53a5144d3bf4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b7d551137fd46e0b3f53a5144d3bf4e" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 35857 } }
I20260807 22:56:48.692382 1690 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5b7d551137fd46e0b3f53a5144d3bf4e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 22:56:48.692538 1690 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5b7d551137fd46e0b3f53a5144d3bf4e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 22:56:48.692740 1690 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5b7d551137fd46e0b3f53a5144d3bf4e [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:56:48.694090 25976 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 22:56:48.695390 1690 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5b7d551137fd46e0b3f53a5144d3bf4e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b7d551137fd46e0b3f53a5144d3bf4e" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 35857 } }
I20260807 22:56:48.695976 1690 leader_election.cc:302] T 00000000000000000000000000000000 P 5b7d551137fd46e0b3f53a5144d3bf4e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5b7d551137fd46e0b3f53a5144d3bf4e; no voters:
I20260807 22:56:48.696697 1690 leader_election.cc:288] T 00000000000000000000000000000000 P 5b7d551137fd46e0b3f53a5144d3bf4e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 22:56:48.696833 1701 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5b7d551137fd46e0b3f53a5144d3bf4e [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:56:48.698912 1690 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5b7d551137fd46e0b3f53a5144d3bf4e [sys.catalog]: configured and running, proceeding with master startup.
I20260807 22:56:48.701480 1701 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5b7d551137fd46e0b3f53a5144d3bf4e [term 1 LEADER]: Becoming Leader. State: Replica: 5b7d551137fd46e0b3f53a5144d3bf4e, State: Running, Role: LEADER
I20260807 22:56:48.702026 1701 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5b7d551137fd46e0b3f53a5144d3bf4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b7d551137fd46e0b3f53a5144d3bf4e" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 35857 } }
I20260807 22:56:48.704651 1702 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b7d551137fd46e0b3f53a5144d3bf4e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5b7d551137fd46e0b3f53a5144d3bf4e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b7d551137fd46e0b3f53a5144d3bf4e" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 35857 } } }
I20260807 22:56:48.705132 1702 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b7d551137fd46e0b3f53a5144d3bf4e [sys.catalog]: This master's current role is: LEADER
I20260807 22:56:48.705447 1705 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b7d551137fd46e0b3f53a5144d3bf4e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5b7d551137fd46e0b3f53a5144d3bf4e. Latest consensus state: current_term: 1 leader_uuid: "5b7d551137fd46e0b3f53a5144d3bf4e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b7d551137fd46e0b3f53a5144d3bf4e" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 35857 } } }
I20260807 22:56:48.705888 1705 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b7d551137fd46e0b3f53a5144d3bf4e [sys.catalog]: This master's current role is: LEADER
I20260807 22:56:48.708282 1712 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 22:56:48.713848 1712 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 22:56:48.724002 1712 catalog_manager.cc:1347] Generated new cluster ID: 0dbdfe84bf3f4e979afb274bb57e27a8
I20260807 22:56:48.724191 1712 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 22:56:48.742368 1712 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 22:56:48.743633 1712 catalog_manager.cc:1504] Loading token signing keys...
I20260807 22:56:48.759323 1712 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5b7d551137fd46e0b3f53a5144d3bf4e: Generated new TSK 0
I20260807 22:56:48.760197 1712 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260807 22:56:48.800098 1741 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260807 22:56:48.800405 1741 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260807 22:56:48.801506 1694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:48.778029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41160 (local address 127.25.94.62:35857)
0807 22:56:48.778225 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:48.778233 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:48.778847 (+ 614us) server_negotiation.cc:407] Connection header received
0807 22:56:48.779190 (+ 343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:48.779196 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:48.779293 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:48.779497 (+ 204us) server_negotiation.cc:226] Negotiated authn=JWT
0807 22:56:48.780056 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:48.781245 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:48.782632 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:48.783140 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:48.783346 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:48.783900 (+ 554us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0807 22:56:48.800768 (+ 16868us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35209/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0807 22:56:48.801201 (+ 433us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41160: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35209/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":63}
W20260807 22:56:48.801642 1694 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41160: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35209/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260807 22:56:48.803112 1740 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:56:48.778011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:35857 (local address 127.0.0.1:41160)
0807 22:56:48.778745 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0807 22:56:48.778761 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 22:56:48.778909 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:56:48.779538 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:56:48.779541 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:56:48.779551 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0807 22:56:48.779902 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:48.779911 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:48.781397 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:48.781409 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:48.782492 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:48.782502 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:48.783403 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:48.783412 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:48.783553 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:48.783609 (+ 56us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0807 22:56:48.801065 (+ 17456us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35209/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0807 22:56:48.801436 (+ 371us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.25.94.62:35857: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35209/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":421,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.25.94.62:35857: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35209/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260807 22:56:48.804862 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 1587
I20260807 22:56:48.806192 1682 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 22:56:48.901485 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 1587
2026-08-07T22:56:48Z chronyd exiting
I20260807 22:56:48.959474 25976 test_util.cc:175] -----------------------------------------------
I20260807 22:56:48.959676 25976 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786143361779964-25976-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (813 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260807 22:56:48.963572 25976 mini_oidc.cc:152] Starting JWKS server
I20260807 22:56:48.965652 25976 webserver.cc:460] Webserver started at https://127.0.0.1:34845/ using document root <none> and password file <none>
I20260807 22:56:48.965907 25976 mini_oidc.cc:189] Starting OIDC Discovery server
I20260807 22:56:48.966640 25976 webserver.cc:460] Webserver started at http://127.0.0.1:32897/ using document root <none> and password file <none>
2026-08-07T22:56:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T22:56:48Z Disabled control of system clock
I20260807 22:56:49.000293 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786143361779964-25976-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786143361779964-25976-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786143361779964-25976-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.62:33777
--webserver_interface=127.25.94.62
--webserver_port=0
--builtin_ntp_servers=127.25.94.20:38719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.94.62:33777
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34845/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786143361779964-25976-0/testchainca.cert with env {}
W20260807 22:56:49.348224 1754 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:56:49.348606 1754 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:56:49.348778 1754 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:56:49.356257 1754 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 22:56:49.356436 1754 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:56:49.356527 1754 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:56:49.356642 1754 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 22:56:49.356729 1754 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260807 22:56:49.356832 1754 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260807 22:56:49.356913 1754 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34845/jwks.json
I20260807 22:56:49.367494 1754 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:38719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786143361779964-25976-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786143361779964-25976-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.94.62:33777
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.62:33777
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34845/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.94.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786143361779964-25976-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786143361779964-25976-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 22:56:49.368927 1754 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:56:49.370306 1754 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:56:49.383239 1766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:56:49.383188 1767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:56:49.383275 1770 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:56:49.384735 1754 server_base.cc:1034] running on GCE node
I20260807 22:56:49.385836 1754 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:56:49.387437 1754 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:56:49.388733 1754 hybrid_clock.cc:648] HybridClock initialized: now 1786143409388670 us; error 58 us; skew 500 ppm
I20260807 22:56:49.389153 1754 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 22:56:49.392069 1754 webserver.cc:460] Webserver started at http://127.25.94.62:42519/ using document root <none> and password file <none>
I20260807 22:56:49.392875 1754 fs_manager.cc:359] Metadata directory not provided
I20260807 22:56:49.393024 1754 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:56:49.393452 1754 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:56:49.396713 1754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786143361779964-25976-0/minicluster-data/master-0/data/instance:
uuid: "5f4bd23ba46245c7b89f5dbc97585798"
format_stamp: "Formatted at 2026-08-07 22:56:49 on dist-test-slave-z8x9"
server_key: "0eb10e6f60a75e207450a2533a8778df"
server_key_iv: "69320c191884d96bdc0d797a2eb63d7e"
server_key_version: "encryptionkey@0"
I20260807 22:56:49.397660 1754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786143361779964-25976-0/minicluster-data/master-0/wal/instance:
uuid: "5f4bd23ba46245c7b89f5dbc97585798"
format_stamp: "Formatted at 2026-08-07 22:56:49 on dist-test-slave-z8x9"
server_key: "0eb10e6f60a75e207450a2533a8778df"
server_key_iv: "69320c191884d96bdc0d797a2eb63d7e"
server_key_version: "encryptionkey@0"
I20260807 22:56:49.403136 1754 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260807 22:56:49.406790 1777 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:56:49.408303 1754 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260807 22:56:49.408507 1754 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786143361779964-25976-0/minicluster-data/master-0/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786143361779964-25976-0/minicluster-data/master-0/wal
uuid: "5f4bd23ba46245c7b89f5dbc97585798"
format_stamp: "Formatted at 2026-08-07 22:56:49 on dist-test-slave-z8x9"
server_key: "0eb10e6f60a75e207450a2533a8778df"
server_key_iv: "69320c191884d96bdc0d797a2eb63d7e"
server_key_version: "encryptionkey@0"
I20260807 22:56:49.408735 1754 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786143361779964-25976-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786143361779964-25976-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786143361779964-25976-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:56:49.428683 1754 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:56:49.429679 1754 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:56:49.430025 1754 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:56:49.450515 1754 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.62:33777
I20260807 22:56:49.450538 1877 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.62:33777 every 8 connection(s)
I20260807 22:56:49.452020 1754 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
I20260807 22:56:49.453981 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 1754
I20260807 22:56:49.454191 25976 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786143361779964-25976-0/minicluster-data/master-0/wal/instance
I20260807 22:56:49.455049 25976 external_mini_cluster.cc:1428] Setting key 249b24454a8d740a5e7a887910ad52f5
I20260807 22:56:49.456835 1878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:56:49.469519 1882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:49.457336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45572 (local address 127.25.94.62:33777)
0807 22:56:49.458404 (+ 1068us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:49.458424 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:49.458467 (+ 43us) server_negotiation.cc:407] Connection header received
0807 22:56:49.460082 (+ 1615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:49.460103 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:49.460545 (+ 442us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:49.460907 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:56:49.461515 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:49.462856 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:49.464660 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:49.465464 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:49.465603 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:49.467028 (+ 1425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:56:49.467036 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:56:49.467049 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 22:56:49.467701 (+ 652us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:56:49.467783 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 22:56:49.467805 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:56:49.467897 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:49.468154 (+ 257us) server_negotiation.cc:299] Negotiation successful
0807 22:56:49.468524 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":649,"thread_start_us":405,"threads_started":1}
I20260807 22:56:49.470697 1878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f4bd23ba46245c7b89f5dbc97585798: Bootstrap starting.
I20260807 22:56:49.473425 25976 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 22:56:49.474162 1878 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5f4bd23ba46245c7b89f5dbc97585798: Neither blocks nor log segments found. Creating new log.
I20260807 22:56:49.475427 1878 log.cc:824] T 00000000000000000000000000000000 P 5f4bd23ba46245c7b89f5dbc97585798: Log is configured to *not* fsync() on all Append() calls
I20260807 22:56:49.478593 1878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f4bd23ba46245c7b89f5dbc97585798: No bootstrap required, opened a new log
I20260807 22:56:49.486796 1878 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5f4bd23ba46245c7b89f5dbc97585798 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f4bd23ba46245c7b89f5dbc97585798" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 33777 } }
I20260807 22:56:49.487206 1878 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5f4bd23ba46245c7b89f5dbc97585798 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:56:49.487324 1878 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5f4bd23ba46245c7b89f5dbc97585798 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f4bd23ba46245c7b89f5dbc97585798, State: Initialized, Role: FOLLOWER
I20260807 22:56:49.487965 1878 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5f4bd23ba46245c7b89f5dbc97585798 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f4bd23ba46245c7b89f5dbc97585798" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 33777 } }
I20260807 22:56:49.488222 1878 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5f4bd23ba46245c7b89f5dbc97585798 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 22:56:49.488380 1878 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5f4bd23ba46245c7b89f5dbc97585798 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 22:56:49.488577 1878 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5f4bd23ba46245c7b89f5dbc97585798 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:56:49.491284 1878 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5f4bd23ba46245c7b89f5dbc97585798 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f4bd23ba46245c7b89f5dbc97585798" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 33777 } }
I20260807 22:56:49.491806 1878 leader_election.cc:302] T 00000000000000000000000000000000 P 5f4bd23ba46245c7b89f5dbc97585798 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5f4bd23ba46245c7b89f5dbc97585798; no voters:
I20260807 22:56:49.492386 1878 leader_election.cc:288] T 00000000000000000000000000000000 P 5f4bd23ba46245c7b89f5dbc97585798 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 22:56:49.492552 1885 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5f4bd23ba46245c7b89f5dbc97585798 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:56:49.493340 1885 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5f4bd23ba46245c7b89f5dbc97585798 [term 1 LEADER]: Becoming Leader. State: Replica: 5f4bd23ba46245c7b89f5dbc97585798, State: Running, Role: LEADER
I20260807 22:56:49.493917 1885 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5f4bd23ba46245c7b89f5dbc97585798 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f4bd23ba46245c7b89f5dbc97585798" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 33777 } }
I20260807 22:56:49.494552 1878 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5f4bd23ba46245c7b89f5dbc97585798 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 22:56:49.496729 1888 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f4bd23ba46245c7b89f5dbc97585798 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5f4bd23ba46245c7b89f5dbc97585798. Latest consensus state: current_term: 1 leader_uuid: "5f4bd23ba46245c7b89f5dbc97585798" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f4bd23ba46245c7b89f5dbc97585798" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 33777 } } }
I20260807 22:56:49.497807 1888 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f4bd23ba46245c7b89f5dbc97585798 [sys.catalog]: This master's current role is: LEADER
I20260807 22:56:49.497382 1886 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f4bd23ba46245c7b89f5dbc97585798 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5f4bd23ba46245c7b89f5dbc97585798" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f4bd23ba46245c7b89f5dbc97585798" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 33777 } } }
I20260807 22:56:49.498418 1886 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f4bd23ba46245c7b89f5dbc97585798 [sys.catalog]: This master's current role is: LEADER
I20260807 22:56:49.500053 1898 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 22:56:49.505137 1898 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 22:56:49.514160 1898 catalog_manager.cc:1347] Generated new cluster ID: 2d6314939675451185b8ff442603a592
I20260807 22:56:49.514292 1898 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 22:56:49.548524 1898 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 22:56:49.550465 1898 catalog_manager.cc:1504] Loading token signing keys...
I20260807 22:56:49.566951 1898 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5f4bd23ba46245c7b89f5dbc97585798: Generated new TSK 0
I20260807 22:56:49.568099 1898 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260807 22:56:49.616127 1928 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260807 22:56:49.616351 1928 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260807 22:56:49.617666 1882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:49.594970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45582 (local address 127.25.94.62:33777)
0807 22:56:49.595232 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:49.595238 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:49.596465 (+ 1227us) server_negotiation.cc:407] Connection header received
0807 22:56:49.596737 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:49.596744 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:49.596844 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:49.597026 (+ 182us) server_negotiation.cc:226] Negotiated authn=JWT
0807 22:56:49.597725 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:49.598899 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:49.600445 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:49.601211 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:49.601346 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:49.601923 (+ 577us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0807 22:56:49.616847 (+ 14924us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34845/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0807 22:56:49.617330 (+ 483us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45582: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34845/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":75}
W20260807 22:56:49.617765 1926 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:56:49.594860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:33777 (local address 127.0.0.1:45582)
0807 22:56:49.596331 (+ 1471us) negotiation.cc:107] Waiting for socket to connect
0807 22:56:49.596352 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 22:56:49.596521 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:56:49.597060 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:56:49.597064 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:56:49.597075 (+ 11us) client_negotiation.cc:189] Negotiated authn=JWT
0807 22:56:49.597472 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:49.597482 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:49.599069 (+ 1587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:49.599074 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:49.600278 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:49.600293 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:49.601402 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:49.601407 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:49.601564 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:49.601603 (+ 39us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0807 22:56:49.617119 (+ 15516us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34845/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0807 22:56:49.617434 (+ 315us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.25.94.62:33777: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34845/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":1222,"thread_start_us":1044,"threads_started":1}
W20260807 22:56:49.617802 1882 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45582: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34845/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260807 22:56:49.619370 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 1754
I20260807 22:56:49.620525 1872 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 22:56:49.717252 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 1754
2026-08-07T22:56:49Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (815 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260807 22:56:49.779140 25976 mini_oidc.cc:152] Starting JWKS server
I20260807 22:56:49.781081 25976 webserver.cc:460] Webserver started at https://127.0.0.1:36117/ using document root <none> and password file <none>
I20260807 22:56:49.781397 25976 mini_oidc.cc:189] Starting OIDC Discovery server
I20260807 22:56:49.782050 25976 webserver.cc:460] Webserver started at http://127.0.0.1:35715/ using document root <none> and password file <none>
2026-08-07T22:56:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T22:56:49Z Disabled control of system clock
I20260807 22:56:49.816115 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786143361779964-25976-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786143361779964-25976-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786143361779964-25976-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.62:41861
--webserver_interface=127.25.94.62
--webserver_port=0
--builtin_ntp_servers=127.25.94.20:41461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.94.62:41861
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36117/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260807 22:56:50.167287 1944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:56:50.167738 1944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:56:50.167920 1944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:56:50.176087 1944 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 22:56:50.176309 1944 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:56:50.176411 1944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:56:50.176532 1944 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 22:56:50.176616 1944 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260807 22:56:50.176712 1944 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260807 22:56:50.176791 1944 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36117/jwks.json
I20260807 22:56:50.187644 1944 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:41461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786143361779964-25976-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786143361779964-25976-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.94.62:41861
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.62:41861
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36117/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.94.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786143361779964-25976-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 22:56:50.189226 1944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:56:50.191179 1944 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:56:50.204720 1955 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:56:50.204802 1962 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:56:50.204741 1956 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:56:50.205962 1944 server_base.cc:1034] running on GCE node
I20260807 22:56:50.206897 1944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:56:50.208441 1944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:56:50.209733 1944 hybrid_clock.cc:648] HybridClock initialized: now 1786143410209677 us; error 44 us; skew 500 ppm
I20260807 22:56:50.210206 1944 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 22:56:50.213294 1944 webserver.cc:460] Webserver started at http://127.25.94.62:38331/ using document root <none> and password file <none>
I20260807 22:56:50.214298 1944 fs_manager.cc:359] Metadata directory not provided
I20260807 22:56:50.214454 1944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:56:50.214959 1944 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:56:50.218297 1944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786143361779964-25976-0/minicluster-data/master-0/data/instance:
uuid: "5505c255998a4774a48490563cdd2ed3"
format_stamp: "Formatted at 2026-08-07 22:56:50 on dist-test-slave-z8x9"
server_key: "cfe9caffe9f2112f7ab5c8998cbc83b0"
server_key_iv: "a3d10385483dc570243e1e79b5f631c3"
server_key_version: "encryptionkey@0"
I20260807 22:56:50.219256 1944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786143361779964-25976-0/minicluster-data/master-0/wal/instance:
uuid: "5505c255998a4774a48490563cdd2ed3"
format_stamp: "Formatted at 2026-08-07 22:56:50 on dist-test-slave-z8x9"
server_key: "cfe9caffe9f2112f7ab5c8998cbc83b0"
server_key_iv: "a3d10385483dc570243e1e79b5f631c3"
server_key_version: "encryptionkey@0"
I20260807 22:56:50.225728 1944 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260807 22:56:50.230042 1970 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:56:50.231914 1944 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260807 22:56:50.232160 1944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786143361779964-25976-0/minicluster-data/master-0/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786143361779964-25976-0/minicluster-data/master-0/wal
uuid: "5505c255998a4774a48490563cdd2ed3"
format_stamp: "Formatted at 2026-08-07 22:56:50 on dist-test-slave-z8x9"
server_key: "cfe9caffe9f2112f7ab5c8998cbc83b0"
server_key_iv: "a3d10385483dc570243e1e79b5f631c3"
server_key_version: "encryptionkey@0"
I20260807 22:56:50.232398 1944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786143361779964-25976-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786143361779964-25976-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786143361779964-25976-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:56:50.248682 1944 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:56:50.249735 1944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:56:50.250082 1944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:56:50.272321 1944 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.62:41861
I20260807 22:56:50.272360 2046 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.62:41861 every 8 connection(s)
I20260807 22:56:50.273995 1944 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
I20260807 22:56:50.278030 2049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:56:50.281095 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 1944
I20260807 22:56:50.281355 25976 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786143361779964-25976-0/minicluster-data/master-0/wal/instance
I20260807 22:56:50.282208 25976 external_mini_cluster.cc:1428] Setting key e5c3e0d5c3d83b05509fe2b3a696a99a
I20260807 22:56:50.291790 2049 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5505c255998a4774a48490563cdd2ed3: Bootstrap starting.
I20260807 22:56:50.295882 2049 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5505c255998a4774a48490563cdd2ed3: Neither blocks nor log segments found. Creating new log.
I20260807 22:56:50.297130 2049 log.cc:824] T 00000000000000000000000000000000 P 5505c255998a4774a48490563cdd2ed3: Log is configured to *not* fsync() on all Append() calls
I20260807 22:56:50.297823 2056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:50.284331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56096 (local address 127.25.94.62:41861)
0807 22:56:50.285500 (+ 1169us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:50.285511 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:50.285558 (+ 47us) server_negotiation.cc:407] Connection header received
0807 22:56:50.287275 (+ 1717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:50.287301 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:50.287833 (+ 532us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:50.288354 (+ 521us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:56:50.288943 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:50.290437 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:50.292469 (+ 2032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:50.293335 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:50.293498 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:50.294930 (+ 1432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:56:50.294938 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:56:50.294954 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 22:56:50.295607 (+ 653us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:56:50.295710 (+ 103us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 22:56:50.295739 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:56:50.295833 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:50.296218 (+ 385us) server_negotiation.cc:299] Negotiation successful
0807 22:56:50.296634 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":653,"thread_start_us":390,"threads_started":1}
I20260807 22:56:50.300395 2049 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5505c255998a4774a48490563cdd2ed3: No bootstrap required, opened a new log
I20260807 22:56:50.302294 25976 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 22:56:50.308493 2049 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5505c255998a4774a48490563cdd2ed3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5505c255998a4774a48490563cdd2ed3" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 41861 } }
I20260807 22:56:50.308905 2049 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5505c255998a4774a48490563cdd2ed3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:56:50.309026 2049 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5505c255998a4774a48490563cdd2ed3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5505c255998a4774a48490563cdd2ed3, State: Initialized, Role: FOLLOWER
I20260807 22:56:50.309703 2049 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5505c255998a4774a48490563cdd2ed3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5505c255998a4774a48490563cdd2ed3" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 41861 } }
I20260807 22:56:50.309964 2049 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5505c255998a4774a48490563cdd2ed3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 22:56:50.310170 2049 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5505c255998a4774a48490563cdd2ed3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 22:56:50.310365 2049 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5505c255998a4774a48490563cdd2ed3 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:56:50.312628 2049 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5505c255998a4774a48490563cdd2ed3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5505c255998a4774a48490563cdd2ed3" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 41861 } }
I20260807 22:56:50.313158 2049 leader_election.cc:302] T 00000000000000000000000000000000 P 5505c255998a4774a48490563cdd2ed3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5505c255998a4774a48490563cdd2ed3; no voters:
I20260807 22:56:50.313942 2049 leader_election.cc:288] T 00000000000000000000000000000000 P 5505c255998a4774a48490563cdd2ed3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 22:56:50.314118 2060 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5505c255998a4774a48490563cdd2ed3 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:56:50.314946 2060 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5505c255998a4774a48490563cdd2ed3 [term 1 LEADER]: Becoming Leader. State: Replica: 5505c255998a4774a48490563cdd2ed3, State: Running, Role: LEADER
I20260807 22:56:50.315555 2060 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5505c255998a4774a48490563cdd2ed3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5505c255998a4774a48490563cdd2ed3" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 41861 } }
I20260807 22:56:50.316265 2049 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5505c255998a4774a48490563cdd2ed3 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 22:56:50.321887 2061 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5505c255998a4774a48490563cdd2ed3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5505c255998a4774a48490563cdd2ed3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5505c255998a4774a48490563cdd2ed3" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 41861 } } }
I20260807 22:56:50.322335 2061 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5505c255998a4774a48490563cdd2ed3 [sys.catalog]: This master's current role is: LEADER
I20260807 22:56:50.322926 2063 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5505c255998a4774a48490563cdd2ed3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5505c255998a4774a48490563cdd2ed3. Latest consensus state: current_term: 1 leader_uuid: "5505c255998a4774a48490563cdd2ed3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5505c255998a4774a48490563cdd2ed3" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 41861 } } }
I20260807 22:56:50.323311 2063 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5505c255998a4774a48490563cdd2ed3 [sys.catalog]: This master's current role is: LEADER
I20260807 22:56:50.328153 2069 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 22:56:50.335427 2069 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 22:56:50.347432 2069 catalog_manager.cc:1347] Generated new cluster ID: 59bbaa330cab454abc24180b55a2c1fc
I20260807 22:56:50.347589 2069 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 22:56:50.382194 2069 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 22:56:50.384161 2069 catalog_manager.cc:1504] Loading token signing keys...
I20260807 22:56:50.394737 2069 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5505c255998a4774a48490563cdd2ed3: Generated new TSK 0
I20260807 22:56:50.395740 2069 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 22:56:50.431550 2056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:50.422384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56102 (local address 127.25.94.62:41861)
0807 22:56:50.422594 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:50.422601 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:50.423013 (+ 412us) server_negotiation.cc:407] Connection header received
0807 22:56:50.423222 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:50.423228 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:50.423324 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:50.423513 (+ 189us) server_negotiation.cc:226] Negotiated authn=JWT
0807 22:56:50.424033 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:50.425216 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:50.426670 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:50.427498 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:50.427635 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:50.428251 (+ 616us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0807 22:56:50.430850 (+ 2599us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36117/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0807 22:56:50.431203 (+ 353us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56102: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36117/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":76}
W20260807 22:56:50.431689 2056 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56102: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36117/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260807 22:56:50.433727 2098 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:56:50.422240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:41861 (local address 127.0.0.1:56102)
0807 22:56:50.422903 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0807 22:56:50.422922 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 22:56:50.423046 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:56:50.423573 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:56:50.423577 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:56:50.423587 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0807 22:56:50.423868 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:50.423876 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:50.425389 (+ 1513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:50.425393 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:50.426510 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:50.426523 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:50.427733 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:50.427739 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:50.427882 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:50.427922 (+ 40us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0807 22:56:50.431099 (+ 3177us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36117/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0807 22:56:50.433217 (+ 2118us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.25.94.62:41861: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36117/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":538,"mutex_wait_us":1739,"thread_start_us":361,"threads_started":1}
I20260807 22:56:50.436136 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 1944
I20260807 22:56:50.437474 2042 generic_service.cc:196] Checking for leaks (request via RPC)
E20260807 22:56:50.446786 2100 webserver.cc:533] Webserver: sslize failed:
E20260807 22:56:50.446940 2100 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260807 22:56:50.535215 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 1944
2026-08-07T22:56:50Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (818 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-07T22:56:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T22:56:50Z Disabled control of system clock
I20260807 22:56:50.631258 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786143361779964-25976-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786143361779964-25976-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786143361779964-25976-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.62:42261
--webserver_interface=127.25.94.62
--webserver_port=0
--builtin_ntp_servers=127.25.94.20:37145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.94.62:42261
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786143361779964-25976-0/testchainca.cert with env {}
W20260807 22:56:50.969100 2107 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:56:50.969465 2107 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:56:50.969671 2107 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:56:50.977007 2107 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 22:56:50.977136 2107 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:56:50.977279 2107 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:56:50.977397 2107 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 22:56:50.977466 2107 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260807 22:56:50.977561 2107 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260807 22:56:50.977622 2107 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260807 22:56:50.988117 2107 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:37145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786143361779964-25976-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786143361779964-25976-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.94.62:42261
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.62:42261
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.94.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786143361779964-25976-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786143361779964-25976-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 22:56:50.989626 2107 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:56:50.990957 2107 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:56:51.003389 2118 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:56:51.003414 2125 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:56:51.003638 2122 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:56:51.004427 2107 server_base.cc:1034] running on GCE node
I20260807 22:56:51.005662 2107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:56:51.007225 2107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:56:51.008448 2107 hybrid_clock.cc:648] HybridClock initialized: now 1786143411008414 us; error 34 us; skew 500 ppm
I20260807 22:56:51.008824 2107 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 22:56:51.011313 2107 webserver.cc:460] Webserver started at http://127.25.94.62:44213/ using document root <none> and password file <none>
I20260807 22:56:51.012120 2107 fs_manager.cc:359] Metadata directory not provided
I20260807 22:56:51.012259 2107 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:56:51.012600 2107 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:56:51.015655 2107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786143361779964-25976-0/minicluster-data/master-0/data/instance:
uuid: "486374b0e2114347aeaa0d267c8a04ca"
format_stamp: "Formatted at 2026-08-07 22:56:51 on dist-test-slave-z8x9"
server_key: "9970a511ed3a9a7316ff967d58cc0c4f"
server_key_iv: "138488489d34c81cfd343dd20240b32e"
server_key_version: "encryptionkey@0"
I20260807 22:56:51.016500 2107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786143361779964-25976-0/minicluster-data/master-0/wal/instance:
uuid: "486374b0e2114347aeaa0d267c8a04ca"
format_stamp: "Formatted at 2026-08-07 22:56:51 on dist-test-slave-z8x9"
server_key: "9970a511ed3a9a7316ff967d58cc0c4f"
server_key_iv: "138488489d34c81cfd343dd20240b32e"
server_key_version: "encryptionkey@0"
I20260807 22:56:51.021852 2107 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260807 22:56:51.025492 2131 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:56:51.026831 2107 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 22:56:51.027022 2107 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786143361779964-25976-0/minicluster-data/master-0/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786143361779964-25976-0/minicluster-data/master-0/wal
uuid: "486374b0e2114347aeaa0d267c8a04ca"
format_stamp: "Formatted at 2026-08-07 22:56:51 on dist-test-slave-z8x9"
server_key: "9970a511ed3a9a7316ff967d58cc0c4f"
server_key_iv: "138488489d34c81cfd343dd20240b32e"
server_key_version: "encryptionkey@0"
I20260807 22:56:51.027256 2107 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786143361779964-25976-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786143361779964-25976-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786143361779964-25976-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:56:51.052067 2107 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:56:51.053037 2107 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:56:51.053385 2107 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:56:51.073715 2107 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.62:42261
I20260807 22:56:51.073815 2203 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.62:42261 every 8 connection(s)
I20260807 22:56:51.075171 2107 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
I20260807 22:56:51.079559 2206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:56:51.083972 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 2107
I20260807 22:56:51.084214 25976 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786143361779964-25976-0/minicluster-data/master-0/wal/instance
I20260807 22:56:51.084885 25976 external_mini_cluster.cc:1428] Setting key b35a8f3bc710b0593cd5bc5772e62665
I20260807 22:56:51.091902 2206 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 486374b0e2114347aeaa0d267c8a04ca: Bootstrap starting.
I20260807 22:56:51.096031 2206 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 486374b0e2114347aeaa0d267c8a04ca: Neither blocks nor log segments found. Creating new log.
I20260807 22:56:51.097594 2206 log.cc:824] T 00000000000000000000000000000000 P 486374b0e2114347aeaa0d267c8a04ca: Log is configured to *not* fsync() on all Append() calls
I20260807 22:56:51.101728 2206 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 486374b0e2114347aeaa0d267c8a04ca: No bootstrap required, opened a new log
I20260807 22:56:51.101837 2217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:51.086821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54744 (local address 127.25.94.62:42261)
0807 22:56:51.087899 (+ 1078us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:51.087911 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:51.087954 (+ 43us) server_negotiation.cc:407] Connection header received
0807 22:56:51.090270 (+ 2316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:51.090298 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:51.090810 (+ 512us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:51.091176 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:56:51.091845 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:51.093327 (+ 1482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:51.095495 (+ 2168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:51.096345 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:51.096521 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:51.098528 (+ 2007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:56:51.098541 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:56:51.098566 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 22:56:51.099429 (+ 863us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:56:51.099519 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 22:56:51.099543 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:56:51.099657 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:51.100049 (+ 392us) server_negotiation.cc:299] Negotiation successful
0807 22:56:51.100461 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":639,"thread_start_us":409,"threads_started":1}
I20260807 22:56:51.107023 25976 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 22:56:51.110911 2206 raft_consensus.cc:348] T 00000000000000000000000000000000 P 486374b0e2114347aeaa0d267c8a04ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "486374b0e2114347aeaa0d267c8a04ca" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 42261 } }
I20260807 22:56:51.111410 2206 raft_consensus.cc:374] T 00000000000000000000000000000000 P 486374b0e2114347aeaa0d267c8a04ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:56:51.111526 2206 raft_consensus.cc:729] T 00000000000000000000000000000000 P 486374b0e2114347aeaa0d267c8a04ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 486374b0e2114347aeaa0d267c8a04ca, State: Initialized, Role: FOLLOWER
I20260807 22:56:51.112110 2206 consensus_queue.cc:260] T 00000000000000000000000000000000 P 486374b0e2114347aeaa0d267c8a04ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "486374b0e2114347aeaa0d267c8a04ca" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 42261 } }
I20260807 22:56:51.112366 2206 raft_consensus.cc:388] T 00000000000000000000000000000000 P 486374b0e2114347aeaa0d267c8a04ca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 22:56:51.112522 2206 raft_consensus.cc:482] T 00000000000000000000000000000000 P 486374b0e2114347aeaa0d267c8a04ca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 22:56:51.112721 2206 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 486374b0e2114347aeaa0d267c8a04ca [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:56:51.115144 2206 raft_consensus.cc:504] T 00000000000000000000000000000000 P 486374b0e2114347aeaa0d267c8a04ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "486374b0e2114347aeaa0d267c8a04ca" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 42261 } }
I20260807 22:56:51.115767 2206 leader_election.cc:302] T 00000000000000000000000000000000 P 486374b0e2114347aeaa0d267c8a04ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 486374b0e2114347aeaa0d267c8a04ca; no voters:
I20260807 22:56:51.116411 2206 leader_election.cc:288] T 00000000000000000000000000000000 P 486374b0e2114347aeaa0d267c8a04ca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 22:56:51.116647 2220 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 486374b0e2114347aeaa0d267c8a04ca [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:56:51.117909 2220 raft_consensus.cc:686] T 00000000000000000000000000000000 P 486374b0e2114347aeaa0d267c8a04ca [term 1 LEADER]: Becoming Leader. State: Replica: 486374b0e2114347aeaa0d267c8a04ca, State: Running, Role: LEADER
I20260807 22:56:51.118607 2220 consensus_queue.cc:237] T 00000000000000000000000000000000 P 486374b0e2114347aeaa0d267c8a04ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "486374b0e2114347aeaa0d267c8a04ca" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 42261 } }
I20260807 22:56:51.118892 2206 sys_catalog.cc:564] T 00000000000000000000000000000000 P 486374b0e2114347aeaa0d267c8a04ca [sys.catalog]: configured and running, proceeding with master startup.
I20260807 22:56:51.121654 2222 sys_catalog.cc:455] T 00000000000000000000000000000000 P 486374b0e2114347aeaa0d267c8a04ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 486374b0e2114347aeaa0d267c8a04ca. Latest consensus state: current_term: 1 leader_uuid: "486374b0e2114347aeaa0d267c8a04ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "486374b0e2114347aeaa0d267c8a04ca" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 42261 } } }
I20260807 22:56:51.122104 2222 sys_catalog.cc:458] T 00000000000000000000000000000000 P 486374b0e2114347aeaa0d267c8a04ca [sys.catalog]: This master's current role is: LEADER
I20260807 22:56:51.122920 2221 sys_catalog.cc:455] T 00000000000000000000000000000000 P 486374b0e2114347aeaa0d267c8a04ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "486374b0e2114347aeaa0d267c8a04ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "486374b0e2114347aeaa0d267c8a04ca" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 42261 } } }
I20260807 22:56:51.123277 2221 sys_catalog.cc:458] T 00000000000000000000000000000000 P 486374b0e2114347aeaa0d267c8a04ca [sys.catalog]: This master's current role is: LEADER
I20260807 22:56:51.125633 2231 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 22:56:51.130844 2231 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 22:56:51.140323 2231 catalog_manager.cc:1347] Generated new cluster ID: 873a563627334135ace3df9a95d2670c
I20260807 22:56:51.140471 2231 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 22:56:51.156497 2231 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 22:56:51.158543 2231 catalog_manager.cc:1504] Loading token signing keys...
I20260807 22:56:51.167080 2231 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 486374b0e2114347aeaa0d267c8a04ca: Generated new TSK 0
I20260807 22:56:51.168171 2231 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 22:56:51.667500 2217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:51.193155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54754 (local address 127.25.94.62:42261)
0807 22:56:51.193412 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:51.193423 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:51.193996 (+ 573us) server_negotiation.cc:407] Connection header received
0807 22:56:51.194279 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:51.194283 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:51.194383 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:51.194557 (+ 174us) server_negotiation.cc:226] Negotiated authn=JWT
0807 22:56:51.195223 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:51.196004 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:51.198325 (+ 2321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:51.198833 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:51.198949 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:51.199726 (+ 777us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0807 22:56:51.666789 (+467063us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0807 22:56:51.667240 (+ 451us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54754: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":83}
W20260807 22:56:51.667671 2217 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54754: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260807 22:56:51.668138 2250 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:56:51.193082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:42261 (local address 127.0.0.1:54754)
0807 22:56:51.193839 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0807 22:56:51.193861 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 22:56:51.194071 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:56:51.194633 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:56:51.194639 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:56:51.194663 (+ 24us) client_negotiation.cc:189] Negotiated authn=JWT
0807 22:56:51.195013 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:51.195028 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:51.196176 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:51.196182 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:51.198046 (+ 1864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:51.198068 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:51.199052 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:51.199059 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:51.199271 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:51.199325 (+ 54us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0807 22:56:51.667124 (+467799us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0807 22:56:51.667427 (+ 303us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.25.94.62:42261: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":592,"thread_start_us":437,"threads_started":1}
I20260807 22:56:51.669305 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 2107
I20260807 22:56:51.670662 2197 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 22:56:51.772867 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 2107
2026-08-07T22:56:51Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (1223 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (430 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (1676 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786143361779964-25976-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 22:56:54 dist-test-slave-z8x9 krb5kdc[2278](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 22:56:54 dist-test-slave-z8x9 krb5kdc[2278](info): set up 2 sockets
Aug 07 22:56:54 dist-test-slave-z8x9 krb5kdc[2278](info): commencing operation
krb5kdc: starting...
W20260807 22:56:55.246702 25976 mini_kdc.cc:121] Time spent starting KDC: real 1.317s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786143361779964-25976-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786143361779964-25976-0/krb5kdc/test-user.keytab.
Aug 07 22:56:55 dist-test-slave-z8x9 krb5kdc[2278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143415, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T22:56:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T22:56:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.94.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.62.keytab.
Entry for principal kudu/127.25.94.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.62.keytab.
I20260807 22:56:55.515029 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786143361779964-25976-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786143361779964-25976-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786143361779964-25976-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.62:43377
--webserver_interface=127.25.94.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.62.keytab
--principal=kudu/127.25.94.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.20:42833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.94.62:43377
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786143361779964-25976-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786143361779964-25976-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786143361779964-25976-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:56:55.853940 2300 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:56:55.854305 2300 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:56:55.854483 2300 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:56:55.861960 2300 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 22:56:55.862143 2300 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:56:55.862259 2300 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:56:55.862372 2300 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 22:56:55.862462 2300 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 22:56:55.872675 2300 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:42833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786143361779964-25976-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786143361779964-25976-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.94.62:43377
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.62.keytab
--principal=kudu/127.25.94.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.62:43377
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.94.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786143361779964-25976-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 22:56:55.874320 2300 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:56:55.875674 2300 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:56:55.888116 2314 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:56:55.888173 2307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:56:55.888149 2306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:56:55.889132 2300 server_base.cc:1034] running on GCE node
I20260807 22:56:55.890074 2300 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:56:55.891537 2300 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:56:55.892783 2300 hybrid_clock.cc:648] HybridClock initialized: now 1786143415892772 us; error 117 us; skew 500 ppm
I20260807 22:56:55.893150 2300 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 22:56:55 dist-test-slave-z8x9 krb5kdc[2278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143415, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:56:55.896486 2300 init.cc:377] Logged in from keytab as kudu/127.25.94.62@KRBTEST.COM (short username kudu)
I20260807 22:56:55.898195 2300 webserver.cc:460] Webserver started at http://127.25.94.62:46161/ using document root <none> and password file <none>
I20260807 22:56:55.898937 2300 fs_manager.cc:359] Metadata directory not provided
I20260807 22:56:55.899096 2300 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:56:55.899454 2300 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:56:55.902642 2300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786143361779964-25976-0/minicluster-data/master-0/data/instance:
uuid: "7947da67670e4362929f08297f836351"
format_stamp: "Formatted at 2026-08-07 22:56:55 on dist-test-slave-z8x9"
server_key: "810ea61a9b277a0bc9e08230483f4643"
server_key_iv: "930e4285d215c2661391add65f2d1658"
server_key_version: "encryptionkey@0"
I20260807 22:56:55.903528 2300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786143361779964-25976-0/minicluster-data/master-0/wal/instance:
uuid: "7947da67670e4362929f08297f836351"
format_stamp: "Formatted at 2026-08-07 22:56:55 on dist-test-slave-z8x9"
server_key: "810ea61a9b277a0bc9e08230483f4643"
server_key_iv: "930e4285d215c2661391add65f2d1658"
server_key_version: "encryptionkey@0"
I20260807 22:56:55.908834 2300 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260807 22:56:55.912513 2322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:56:55.914031 2300 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 22:56:55.914335 2300 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786143361779964-25976-0/minicluster-data/master-0/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786143361779964-25976-0/minicluster-data/master-0/wal
uuid: "7947da67670e4362929f08297f836351"
format_stamp: "Formatted at 2026-08-07 22:56:55 on dist-test-slave-z8x9"
server_key: "810ea61a9b277a0bc9e08230483f4643"
server_key_iv: "930e4285d215c2661391add65f2d1658"
server_key_version: "encryptionkey@0"
I20260807 22:56:55.914541 2300 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786143361779964-25976-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786143361779964-25976-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786143361779964-25976-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:56:55.939525 2300 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:56:55.944182 2300 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:56:55.944500 2300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:56:55.963866 2300 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.62:43377
I20260807 22:56:55.963904 2407 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.62:43377 every 8 connection(s)
I20260807 22:56:55.965308 2300 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
I20260807 22:56:55.969030 2408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:56:55.969080 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 2300
I20260807 22:56:55.969352 25976 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786143361779964-25976-0/minicluster-data/master-0/wal/instance
I20260807 22:56:55.970041 25976 external_mini_cluster.cc:1428] Setting key ab248c30b10d5021e3caa81a62156c69
I20260807 22:56:55.983636 2408 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7947da67670e4362929f08297f836351: Bootstrap starting.
Aug 07 22:56:55 dist-test-slave-z8x9 krb5kdc[2278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143415, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.62@KRBTEST.COM
I20260807 22:56:55.987764 2408 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7947da67670e4362929f08297f836351: Neither blocks nor log segments found. Creating new log.
I20260807 22:56:55.989326 2408 log.cc:824] T 00000000000000000000000000000000 P 7947da67670e4362929f08297f836351: Log is configured to *not* fsync() on all Append() calls
I20260807 22:56:55.992861 2408 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7947da67670e4362929f08297f836351: No bootstrap required, opened a new log
I20260807 22:56:55.996124 2413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:55.972092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37242 (local address 127.25.94.62:43377)
0807 22:56:55.972893 (+ 801us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:55.972908 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:55.972950 (+ 42us) server_negotiation.cc:407] Connection header received
0807 22:56:55.975363 (+ 2413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:55.975390 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:55.975805 (+ 415us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:55.976152 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:56:55.978059 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:55.979379 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:55.981283 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:55.981963 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:55.982108 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:55.986960 (+ 4852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:56:55.986972 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:56:55.986992 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:56:55.987042 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:56:55.991127 (+ 4085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:55.991937 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:55.991948 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:55.991956 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:55.992178 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:55.992677 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:55.992683 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:55.992686 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:55.993470 (+ 784us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:56:55.993793 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:55.994179 (+ 386us) server_negotiation.cc:299] Negotiation successful
0807 22:56:55.994646 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":588,"thread_start_us":379,"threads_started":1}
I20260807 22:56:56.002689 2408 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7947da67670e4362929f08297f836351 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7947da67670e4362929f08297f836351" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 43377 } }
I20260807 22:56:56.003103 2408 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7947da67670e4362929f08297f836351 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:56:56.003247 2408 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7947da67670e4362929f08297f836351 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7947da67670e4362929f08297f836351, State: Initialized, Role: FOLLOWER
I20260807 22:56:56.003906 2408 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7947da67670e4362929f08297f836351 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7947da67670e4362929f08297f836351" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 43377 } }
I20260807 22:56:56.004185 2408 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7947da67670e4362929f08297f836351 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 22:56:56.004379 2408 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7947da67670e4362929f08297f836351 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 22:56:56.004552 2408 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7947da67670e4362929f08297f836351 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:56:56.006865 2408 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7947da67670e4362929f08297f836351 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7947da67670e4362929f08297f836351" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 43377 } }
I20260807 22:56:56.007372 2408 leader_election.cc:302] T 00000000000000000000000000000000 P 7947da67670e4362929f08297f836351 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7947da67670e4362929f08297f836351; no voters:
I20260807 22:56:56.007942 2408 leader_election.cc:288] T 00000000000000000000000000000000 P 7947da67670e4362929f08297f836351 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 22:56:56.008113 2415 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7947da67670e4362929f08297f836351 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:56:56.009094 2415 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7947da67670e4362929f08297f836351 [term 1 LEADER]: Becoming Leader. State: Replica: 7947da67670e4362929f08297f836351, State: Running, Role: LEADER
I20260807 22:56:56.009859 2415 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7947da67670e4362929f08297f836351 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7947da67670e4362929f08297f836351" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 43377 } }
I20260807 22:56:56.010123 2408 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7947da67670e4362929f08297f836351 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 22:56:56.012792 2417 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7947da67670e4362929f08297f836351 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7947da67670e4362929f08297f836351. Latest consensus state: current_term: 1 leader_uuid: "7947da67670e4362929f08297f836351" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7947da67670e4362929f08297f836351" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 43377 } } }
I20260807 22:56:56.013368 2417 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7947da67670e4362929f08297f836351 [sys.catalog]: This master's current role is: LEADER
I20260807 22:56:56.014031 2416 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7947da67670e4362929f08297f836351 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7947da67670e4362929f08297f836351" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7947da67670e4362929f08297f836351" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 43377 } } }
I20260807 22:56:56.014396 2416 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7947da67670e4362929f08297f836351 [sys.catalog]: This master's current role is: LEADER
I20260807 22:56:56.016360 2429 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 22:56:56.021754 2429 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 22:56:56.031584 2429 catalog_manager.cc:1347] Generated new cluster ID: 41a32ea996004c669c08039f8cccbedb
I20260807 22:56:56.031724 2429 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 22:56:56.057583 2429 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 22:56:56.059180 2429 catalog_manager.cc:1504] Loading token signing keys...
I20260807 22:56:56.068905 2429 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7947da67670e4362929f08297f836351: Generated new TSK 0
I20260807 22:56:56.069851 2429 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.94.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.1.keytab.
Entry for principal kudu/127.25.94.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.1.keytab.
I20260807 22:56:56.138232 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786143361779964-25976-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786143361779964-25976-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786143361779964-25976-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786143361779964-25976-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.1:0
--local_ip_for_outbound_sockets=127.25.94.1
--webserver_interface=127.25.94.1
--webserver_port=0
--tserver_master_addrs=127.25.94.62:43377
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.1.keytab
--principal=kudu/127.25.94.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.20:42833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786143361779964-25976-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786143361779964-25976-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786143361779964-25976-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:56:56.519387 2451 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:56:56.519719 2451 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:56:56.519910 2451 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:56:56.527153 2451 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:56:56.527261 2451 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:56:56.527467 2451 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.1
I20260807 22:56:56.537374 2451 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:42833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786143361779964-25976-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786143361779964-25976-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.1.keytab
--principal=kudu/127.25.94.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786143361779964-25976-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.94.1
--webserver_port=0
--tserver_master_addrs=127.25.94.62:43377
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.1
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786143361779964-25976-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 22:56:56.538919 2451 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:56:56.540333 2451 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:56:56.554564 2459 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:56:56.554533 2460 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:56:56.554728 2462 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:56:56.555399 2451 server_base.cc:1034] running on GCE node
I20260807 22:56:56.556079 2451 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:56:56.557060 2451 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:56:56.558337 2451 hybrid_clock.cc:648] HybridClock initialized: now 1786143416558274 us; error 53 us; skew 500 ppm
I20260807 22:56:56.558708 2451 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 22:56:56 dist-test-slave-z8x9 krb5kdc[2278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143416, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:56:56.562060 2451 init.cc:377] Logged in from keytab as kudu/127.25.94.1@KRBTEST.COM (short username kudu)
I20260807 22:56:56.563620 2451 webserver.cc:460] Webserver started at http://127.25.94.1:41603/ using document root <none> and password file <none>
I20260807 22:56:56.564271 2451 fs_manager.cc:359] Metadata directory not provided
I20260807 22:56:56.564374 2451 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:56:56.564635 2451 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:56:56.567540 2451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786143361779964-25976-0/minicluster-data/ts-0/data/instance:
uuid: "2895e609f8814e779bd0db9a537492eb"
format_stamp: "Formatted at 2026-08-07 22:56:56 on dist-test-slave-z8x9"
server_key: "be1fe5128a4eefde427837623b0ebd3b"
server_key_iv: "30d34d4ce4c80e0f3e3fe73955833c06"
server_key_version: "encryptionkey@0"
I20260807 22:56:56.568423 2451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786143361779964-25976-0/minicluster-data/ts-0/wal/instance:
uuid: "2895e609f8814e779bd0db9a537492eb"
format_stamp: "Formatted at 2026-08-07 22:56:56 on dist-test-slave-z8x9"
server_key: "be1fe5128a4eefde427837623b0ebd3b"
server_key_iv: "30d34d4ce4c80e0f3e3fe73955833c06"
server_key_version: "encryptionkey@0"
I20260807 22:56:56.573861 2451 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260807 22:56:56.577471 2475 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:56:56.578800 2451 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260807 22:56:56.578984 2451 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786143361779964-25976-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786143361779964-25976-0/minicluster-data/ts-0/wal
uuid: "2895e609f8814e779bd0db9a537492eb"
format_stamp: "Formatted at 2026-08-07 22:56:56 on dist-test-slave-z8x9"
server_key: "be1fe5128a4eefde427837623b0ebd3b"
server_key_iv: "30d34d4ce4c80e0f3e3fe73955833c06"
server_key_version: "encryptionkey@0"
I20260807 22:56:56.579186 2451 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786143361779964-25976-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786143361779964-25976-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786143361779964-25976-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:56:56.595413 2451 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:56:56.600256 2451 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:56:56.600572 2451 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:56:56.601598 2451 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:56:56.602974 2451 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:56:56.603049 2451 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:56:56.603139 2451 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:56:56.603226 2451 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:56:56.638916 2451 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.1:46849
I20260807 22:56:56.638939 2655 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.1:46849 every 8 connection(s)
I20260807 22:56:56.640259 2451 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786143361779964-25976-0/minicluster-data/ts-0/data/info.pb
I20260807 22:56:56.640736 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 2451
I20260807 22:56:56.640990 25976 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786143361779964-25976-0/minicluster-data/ts-0/wal/instance
I20260807 22:56:56.641852 25976 external_mini_cluster.cc:1428] Setting key 9435cf38a064c5f468521d4811249711
Aug 07 22:56:56 dist-test-slave-z8x9 krb5kdc[2278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143416, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.1@KRBTEST.COM for kudu/127.25.94.62@KRBTEST.COM
I20260807 22:56:56.663921 2660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:56.643596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:43377 (local address 127.25.94.1:47041)
0807 22:56:56.644438 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0807 22:56:56.644495 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0807 22:56:56.646294 (+ 1799us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:56:56.646979 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:56:56.646987 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:56:56.647559 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:56:56.648100 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:56.648117 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:56.649636 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:56.649639 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:56.650885 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:56.650897 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:56.651747 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:56.651756 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:56.652081 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:56.652902 (+ 821us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:56:56.652930 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:56:56.655421 (+ 2491us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:56:56.658771 (+ 3350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:56:56.658780 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:56:56.658796 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:56:56.659240 (+ 444us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:56:56.659643 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:56:56.659648 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:56:56.659651 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:56:56.659932 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:56:56.660733 (+ 801us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:56:56.660780 (+ 47us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:56:56.661341 (+ 561us) client_negotiation.cc:769] Sending connection context
0807 22:56:56.661696 (+ 355us) client_negotiation.cc:240] Negotiation successful
0807 22:56:56.662340 (+ 644us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":526,"threads_started":1}
I20260807 22:56:56.665072 2662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:56.644064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.1:47041 (local address 127.25.94.62:43377)
0807 22:56:56.644537 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:56.644543 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:56.644572 (+ 29us) server_negotiation.cc:407] Connection header received
0807 22:56:56.646563 (+ 1991us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:56.646571 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:56.646664 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:56.646798 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:56:56.648278 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:56.649478 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:56.651045 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:56.651496 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:56.651594 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:56.655657 (+ 4063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:56:56.655662 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:56:56.655665 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:56:56.655699 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:56:56.658610 (+ 2911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:56.659382 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:56.659388 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:56.659392 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:56.659524 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:56.660082 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:56.660091 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:56.660097 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:56.660562 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:56:56.660752 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:56.664474 (+ 3722us) server_negotiation.cc:299] Negotiation successful
0807 22:56:56.664720 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":871,"threads_started":1}
WARNING: no policy specified for kudu/127.25.94.2@KRBTEST.COM; defaulting to no policy
I20260807 22:56:56.667874 2656 heartbeater.cc:347] Connected to a master server at 127.25.94.62:43377
I20260807 22:56:56.668255 2656 heartbeater.cc:464] Registering TS with master...
I20260807 22:56:56.669304 2656 heartbeater.cc:511] Master 127.25.94.62:43377 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.2@KRBTEST.COM" created.
I20260807 22:56:56.672369 2353 ts_manager.cc:194] Registered new tserver with Master: 2895e609f8814e779bd0db9a537492eb (127.25.94.1:46849)
I20260807 22:56:56.674010 2353 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.94.1@KRBTEST.COM'} at 127.25.94.1:47041
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.2.keytab.
Entry for principal kudu/127.25.94.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.2.keytab.
I20260807 22:56:56.701439 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786143361779964-25976-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786143361779964-25976-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786143361779964-25976-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786143361779964-25976-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.2:0
--local_ip_for_outbound_sockets=127.25.94.2
--webserver_interface=127.25.94.2
--webserver_port=0
--tserver_master_addrs=127.25.94.62:43377
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.2.keytab
--principal=kudu/127.25.94.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.20:42833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786143361779964-25976-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786143361779964-25976-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786143361779964-25976-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:56:57.035584 2665 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:56:57.036080 2665 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:56:57.036288 2665 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:56:57.044032 2665 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:56:57.044206 2665 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:56:57.044404 2665 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.2
I20260807 22:56:57.054598 2665 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:42833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786143361779964-25976-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786143361779964-25976-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.2.keytab
--principal=kudu/127.25.94.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786143361779964-25976-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.94.2
--webserver_port=0
--tserver_master_addrs=127.25.94.62:43377
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.2
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786143361779964-25976-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 22:56:57.056041 2665 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:56:57.057488 2665 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:56:57.071502 2679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:56:57.071502 2673 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:56:57.071587 2672 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:56:57.072752 2665 server_base.cc:1034] running on GCE node
I20260807 22:56:57.073556 2665 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:56:57.074612 2665 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:56:57.075867 2665 hybrid_clock.cc:648] HybridClock initialized: now 1786143417075814 us; error 46 us; skew 500 ppm
I20260807 22:56:57.076370 2665 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 22:56:57 dist-test-slave-z8x9 krb5kdc[2278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143417, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:56:57.080021 2665 init.cc:377] Logged in from keytab as kudu/127.25.94.2@KRBTEST.COM (short username kudu)
I20260807 22:56:57.081754 2665 webserver.cc:460] Webserver started at http://127.25.94.2:40227/ using document root <none> and password file <none>
I20260807 22:56:57.082646 2665 fs_manager.cc:359] Metadata directory not provided
I20260807 22:56:57.082819 2665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:56:57.083109 2665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:56:57.086426 2665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786143361779964-25976-0/minicluster-data/ts-1/data/instance:
uuid: "b0b99164a01c4e09a81713420ce02c45"
format_stamp: "Formatted at 2026-08-07 22:56:57 on dist-test-slave-z8x9"
server_key: "62c92fde0b483fa60703815369c097a6"
server_key_iv: "ea47b921582ebb24206519cc98a9b6dd"
server_key_version: "encryptionkey@0"
I20260807 22:56:57.087250 2665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786143361779964-25976-0/minicluster-data/ts-1/wal/instance:
uuid: "b0b99164a01c4e09a81713420ce02c45"
format_stamp: "Formatted at 2026-08-07 22:56:57 on dist-test-slave-z8x9"
server_key: "62c92fde0b483fa60703815369c097a6"
server_key_iv: "ea47b921582ebb24206519cc98a9b6dd"
server_key_version: "encryptionkey@0"
I20260807 22:56:57.093451 2665 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260807 22:56:57.097456 2687 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:56:57.098977 2665 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 22:56:57.099139 2665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786143361779964-25976-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786143361779964-25976-0/minicluster-data/ts-1/wal
uuid: "b0b99164a01c4e09a81713420ce02c45"
format_stamp: "Formatted at 2026-08-07 22:56:57 on dist-test-slave-z8x9"
server_key: "62c92fde0b483fa60703815369c097a6"
server_key_iv: "ea47b921582ebb24206519cc98a9b6dd"
server_key_version: "encryptionkey@0"
I20260807 22:56:57.099359 2665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786143361779964-25976-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786143361779964-25976-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786143361779964-25976-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:56:57.125699 2665 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:56:57.130614 2665 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:56:57.130960 2665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:56:57.132073 2665 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:56:57.133884 2665 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:56:57.133972 2665 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:56:57.134109 2665 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:56:57.134186 2665 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:56:57.172537 2665 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.2:38137
I20260807 22:56:57.172593 2879 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.2:38137 every 8 connection(s)
I20260807 22:56:57.174031 2665 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786143361779964-25976-0/minicluster-data/ts-1/data/info.pb
I20260807 22:56:57.175017 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 2665
I20260807 22:56:57.175228 25976 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786143361779964-25976-0/minicluster-data/ts-1/wal/instance
I20260807 22:56:57.175805 25976 external_mini_cluster.cc:1428] Setting key 48e305f42162158c2d29ab7943eabd8c
Aug 07 22:56:57 dist-test-slave-z8x9 krb5kdc[2278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143417, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.2@KRBTEST.COM for kudu/127.25.94.62@KRBTEST.COM
I20260807 22:56:57.200682 2882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:57.177334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.2:36699 (local address 127.25.94.62:43377)
0807 22:56:57.177814 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:57.177820 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:57.179158 (+ 1338us) server_negotiation.cc:407] Connection header received
0807 22:56:57.182315 (+ 3157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:57.182333 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:57.182444 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:57.182604 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:56:57.184941 (+ 2337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:57.185930 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:57.188177 (+ 2247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:57.188938 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:57.189096 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:57.192703 (+ 3607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:56:57.192712 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:56:57.192720 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:56:57.192769 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:56:57.196650 (+ 3881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:57.197564 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:57.197573 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:57.197580 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:57.197749 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:57.198394 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:57.198400 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:57.198405 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:57.198837 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:56:57.199053 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:57.199987 (+ 934us) server_negotiation.cc:299] Negotiation successful
0807 22:56:57.200252 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":233,"threads_started":1}
I20260807 22:56:57.201486 2883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:57.177995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:43377 (local address 127.25.94.2:36699)
0807 22:56:57.178971 (+ 976us) negotiation.cc:107] Waiting for socket to connect
0807 22:56:57.179020 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0807 22:56:57.181851 (+ 2831us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:56:57.182899 (+ 1048us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:56:57.182914 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:56:57.183819 (+ 905us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:56:57.184705 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:57.184744 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:57.186112 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:57.186120 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:57.187994 (+ 1874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:57.188007 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:57.189158 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:57.189163 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:57.189436 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:57.190181 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:56:57.190205 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:56:57.192451 (+ 2246us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:56:57.196827 (+ 4376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:56:57.196834 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:56:57.196847 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:56:57.197429 (+ 582us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:56:57.197947 (+ 518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:56:57.197954 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:56:57.197960 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:56:57.198240 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:56:57.199027 (+ 787us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:56:57.199033 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:56:57.199549 (+ 516us) client_negotiation.cc:769] Sending connection context
0807 22:56:57.199808 (+ 259us) client_negotiation.cc:240] Negotiation successful
0807 22:56:57.200240 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":672,"thread_start_us":389,"threads_started":1}
WARNING: no policy specified for kudu/127.25.94.3@KRBTEST.COM; defaulting to no policy
I20260807 22:56:57.203162 2880 heartbeater.cc:347] Connected to a master server at 127.25.94.62:43377
I20260807 22:56:57.203569 2880 heartbeater.cc:464] Registering TS with master...
I20260807 22:56:57.204469 2880 heartbeater.cc:511] Master 127.25.94.62:43377 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.3@KRBTEST.COM" created.
I20260807 22:56:57.206316 2353 ts_manager.cc:194] Registered new tserver with Master: b0b99164a01c4e09a81713420ce02c45 (127.25.94.2:38137)
I20260807 22:56:57.207672 2353 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.94.2@KRBTEST.COM'} at 127.25.94.2:36699
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.3.keytab.
Entry for principal kudu/127.25.94.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.3.keytab.
I20260807 22:56:57.238193 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786143361779964-25976-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786143361779964-25976-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786143361779964-25976-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786143361779964-25976-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.3:0
--local_ip_for_outbound_sockets=127.25.94.3
--webserver_interface=127.25.94.3
--webserver_port=0
--tserver_master_addrs=127.25.94.62:43377
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.3.keytab
--principal=kudu/127.25.94.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.20:42833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786143361779964-25976-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786143361779964-25976-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786143361779964-25976-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:56:57.577890 2888 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:56:57.578239 2888 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:56:57.578418 2888 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:56:57.586118 2888 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:56:57.586282 2888 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:56:57.586469 2888 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.3
I20260807 22:56:57.597029 2888 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:42833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786143361779964-25976-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786143361779964-25976-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.3.keytab
--principal=kudu/127.25.94.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786143361779964-25976-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.94.3
--webserver_port=0
--tserver_master_addrs=127.25.94.62:43377
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.3
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786143361779964-25976-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 22:56:57.598519 2888 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:56:57.599926 2888 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:56:57.614032 2895 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:56:57.614105 2898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:56:57.615845 2893 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:56:57.616394 2888 server_base.cc:1034] running on GCE node
I20260807 22:56:57.617018 2888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:56:57.618032 2888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:56:57.619295 2888 hybrid_clock.cc:648] HybridClock initialized: now 1786143417619235 us; error 44 us; skew 500 ppm
I20260807 22:56:57.619642 2888 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 22:56:57 dist-test-slave-z8x9 krb5kdc[2278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143417, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:56:57.622934 2888 init.cc:377] Logged in from keytab as kudu/127.25.94.3@KRBTEST.COM (short username kudu)
I20260807 22:56:57.624483 2888 webserver.cc:460] Webserver started at http://127.25.94.3:41357/ using document root <none> and password file <none>
I20260807 22:56:57.625216 2888 fs_manager.cc:359] Metadata directory not provided
I20260807 22:56:57.625370 2888 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:56:57.625707 2888 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:56:57.628574 2888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786143361779964-25976-0/minicluster-data/ts-2/data/instance:
uuid: "52d7726bc2c3463480143ae990561aca"
format_stamp: "Formatted at 2026-08-07 22:56:57 on dist-test-slave-z8x9"
server_key: "0d6a9e799a74cfe71deb36cc27979c54"
server_key_iv: "67138f5e04f5c5ecdd2f591102283eb8"
server_key_version: "encryptionkey@0"
I20260807 22:56:57.629446 2888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786143361779964-25976-0/minicluster-data/ts-2/wal/instance:
uuid: "52d7726bc2c3463480143ae990561aca"
format_stamp: "Formatted at 2026-08-07 22:56:57 on dist-test-slave-z8x9"
server_key: "0d6a9e799a74cfe71deb36cc27979c54"
server_key_iv: "67138f5e04f5c5ecdd2f591102283eb8"
server_key_version: "encryptionkey@0"
I20260807 22:56:57.635135 2888 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260807 22:56:57.639880 2911 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:56:57.641476 2888 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260807 22:56:57.641664 2888 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786143361779964-25976-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786143361779964-25976-0/minicluster-data/ts-2/wal
uuid: "52d7726bc2c3463480143ae990561aca"
format_stamp: "Formatted at 2026-08-07 22:56:57 on dist-test-slave-z8x9"
server_key: "0d6a9e799a74cfe71deb36cc27979c54"
server_key_iv: "67138f5e04f5c5ecdd2f591102283eb8"
server_key_version: "encryptionkey@0"
I20260807 22:56:57.641901 2888 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786143361779964-25976-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786143361779964-25976-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786143361779964-25976-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:56:57.671028 2888 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:56:57.677140 2656 heartbeater.cc:503] Master 127.25.94.62:43377 was elected leader, sending a full tablet report...
I20260807 22:56:57.677706 2888 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:56:57.678020 2888 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:56:57.679145 2888 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:56:57.680635 2888 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:56:57.680721 2888 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:56:57.680809 2888 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:56:57.680855 2888 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:56:57.717612 2888 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.3:42093
I20260807 22:56:57.717656 3083 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.3:42093 every 8 connection(s)
I20260807 22:56:57.719066 2888 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786143361779964-25976-0/minicluster-data/ts-2/data/info.pb
I20260807 22:56:57.723613 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 2888
I20260807 22:56:57.723845 25976 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786143361779964-25976-0/minicluster-data/ts-2/wal/instance
I20260807 22:56:57.724337 25976 external_mini_cluster.cc:1428] Setting key 2740b453b05ee5cd37c11ce60dbdb67e
Aug 07 22:56:57 dist-test-slave-z8x9 krb5kdc[2278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143417, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.3@KRBTEST.COM for kudu/127.25.94.62@KRBTEST.COM
I20260807 22:56:57.740958 3089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:57.721867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.3:42671 (local address 127.25.94.62:43377)
0807 22:56:57.722306 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:57.722310 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:57.723389 (+ 1079us) server_negotiation.cc:407] Connection header received
0807 22:56:57.725647 (+ 2258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:57.725653 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:57.725732 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:57.725849 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:56:57.727607 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:57.728375 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:57.730054 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:57.730454 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:57.730589 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:57.734143 (+ 3554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:56:57.734148 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:56:57.734152 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:56:57.734187 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:56:57.737397 (+ 3210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:57.738272 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:57.738276 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:57.738279 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:57.738456 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:57.738998 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:57.739010 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:57.739012 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:57.739427 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:56:57.739598 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:57.740350 (+ 752us) server_negotiation.cc:299] Negotiation successful
0807 22:56:57.740585 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":213,"threads_started":1}
I20260807 22:56:57.742035 3090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:57.722481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:43377 (local address 127.25.94.3:42671)
0807 22:56:57.723236 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0807 22:56:57.723274 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0807 22:56:57.725201 (+ 1927us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:56:57.726100 (+ 899us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:56:57.726108 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:56:57.726766 (+ 658us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:56:57.727443 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:57.727460 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:57.728541 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:57.728545 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:57.729907 (+ 1362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:56:57.729920 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:56:57.730683 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:56:57.730690 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:56:57.730948 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:57.731686 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:56:57.731710 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:56:57.733920 (+ 2210us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:56:57.737582 (+ 3662us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:56:57.737591 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:56:57.737608 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:56:57.738114 (+ 506us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:56:57.738609 (+ 495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:56:57.738617 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:56:57.738622 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:56:57.738868 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:56:57.739567 (+ 699us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:56:57.739576 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:56:57.739948 (+ 372us) client_negotiation.cc:769] Sending connection context
0807 22:56:57.740227 (+ 279us) client_negotiation.cc:240] Negotiation successful
0807 22:56:57.740807 (+ 580us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":306,"threads_started":1}
I20260807 22:56:57.743372 3084 heartbeater.cc:347] Connected to a master server at 127.25.94.62:43377
I20260807 22:56:57.743685 3084 heartbeater.cc:464] Registering TS with master...
I20260807 22:56:57.744400 3084 heartbeater.cc:511] Master 127.25.94.62:43377 requested a full tablet report, sending...
I20260807 22:56:57.745934 2353 ts_manager.cc:194] Registered new tserver with Master: 52d7726bc2c3463480143ae990561aca (127.25.94.3:42093)
I20260807 22:56:57.746753 2353 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.94.3@KRBTEST.COM'} at 127.25.94.3:42671
I20260807 22:56:57.752457 25976 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786143361779964-25976-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786143361779964-25976-0/krb5kdc/test-admin.keytab.
Aug 07 22:56:57 dist-test-slave-z8x9 krb5kdc[2278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143417, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:56:57.783350 25976 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260807 22:56:57.788687 3096 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:56:57.786209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53206 (local address 127.25.94.1:46849)
0807 22:56:57.786893 (+ 684us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:57.786901 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:57.786927 (+ 26us) server_negotiation.cc:407] Connection header received
0807 22:56:57.786990 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:57.786996 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:57.787199 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:57.787335 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:56:57.788314 (+ 979us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53206: BlockingRecv error: recv got EOF from 127.0.0.1:53206 (error 108)
Metrics: {"server-negotiator.queue_time_us":499,"thread_start_us":348,"threads_started":1}
W20260807 22:56:57.788748 25976 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786143361779964-25976-0/krb5kdc/krb5cc' of type FILE
Aug 07 22:56:57 dist-test-slave-z8x9 krb5kdc[2278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143417, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:56:57.790693 25976 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 07 22:56:57 dist-test-slave-z8x9 krb5kdc[2278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143417, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:56:57.792146 25976 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 07 22:56:57 dist-test-slave-z8x9 krb5kdc[2278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143417, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.1@KRBTEST.COM
I20260807 22:56:57.808693 3096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:57.793955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53220 (local address 127.25.94.1:46849)
0807 22:56:57.794103 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:57.794108 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:57.794410 (+ 302us) server_negotiation.cc:407] Connection header received
0807 22:56:57.794554 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:57.794557 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:57.794614 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:57.794693 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:56:57.795673 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:57.796648 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:57.798017 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:57.798684 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:57.798809 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:57.802749 (+ 3940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:56:57.802763 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:56:57.802778 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:56:57.802815 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:56:57.805995 (+ 3180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:57.806684 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:57.806691 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:57.806694 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:57.806818 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:57.807127 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:57.807130 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:57.807132 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:57.807576 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:56:57.807921 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:57.808124 (+ 203us) server_negotiation.cc:299] Negotiation successful
0807 22:56:57.808408 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260807 22:56:57.814924 3096 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:56:57.813095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53224 (local address 127.25.94.1:46849)
0807 22:56:57.813333 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:57.813341 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:57.813494 (+ 153us) server_negotiation.cc:407] Connection header received
0807 22:56:57.813619 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:57.813622 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:57.813692 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:57.813772 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:56:57.814690 (+ 918us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53224: BlockingRecv error: recv got EOF from 127.0.0.1:53224 (error 108)
Metrics: {"server-negotiator.queue_time_us":104}
W20260807 22:56:57.815116 25976 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786143361779964-25976-0/krb5kdc/krb5cc' of type FILE
Aug 07 22:56:57 dist-test-slave-z8x9 krb5kdc[2278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143417, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:56:57.816843 25976 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 07 22:56:57 dist-test-slave-z8x9 krb5kdc[2278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143417, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:56:57.818410 25976 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 07 22:56:57 dist-test-slave-z8x9 krb5kdc[2278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143417, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.1@KRBTEST.COM
I20260807 22:56:57.833467 3096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:57.819975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53238 (local address 127.25.94.1:46849)
0807 22:56:57.820185 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:57.820194 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:57.820394 (+ 200us) server_negotiation.cc:407] Connection header received
0807 22:56:57.820580 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:57.820586 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:57.820650 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:57.820755 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:56:57.821861 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:57.822583 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:57.824055 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:57.824561 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:57.824704 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:57.828577 (+ 3873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:56:57.828583 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:56:57.828587 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:56:57.828632 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:56:57.831213 (+ 2581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:57.831822 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:57.831826 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:57.831828 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:57.831952 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:57.832291 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:57.832294 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:57.832296 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:57.832575 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:56:57.832776 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:57.833006 (+ 230us) server_negotiation.cc:299] Negotiation successful
0807 22:56:57.833160 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260807 22:56:57.838092 3096 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:56:57.836253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53246 (local address 127.25.94.1:46849)
0807 22:56:57.836489 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:57.836496 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:57.836748 (+ 252us) server_negotiation.cc:407] Connection header received
0807 22:56:57.836929 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:57.836934 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:57.837008 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:57.837110 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:56:57.837872 (+ 762us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53246: BlockingRecv error: recv got EOF from 127.0.0.1:53246 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
W20260807 22:56:57.838459 25976 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786143361779964-25976-0/krb5kdc/krb5cc' of type FILE
Aug 07 22:56:57 dist-test-slave-z8x9 krb5kdc[2278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143417, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:56:57.840015 25976 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 07 22:56:57 dist-test-slave-z8x9 krb5kdc[2278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143417, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:56:57.841410 25976 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 07 22:56:57 dist-test-slave-z8x9 krb5kdc[2278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143417, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.1@KRBTEST.COM
I20260807 22:56:57.855623 3096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:57.842681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53258 (local address 127.25.94.1:46849)
0807 22:56:57.842890 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:57.842896 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:57.843075 (+ 179us) server_negotiation.cc:407] Connection header received
0807 22:56:57.843201 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:57.843205 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:57.843262 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:57.843337 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:56:57.844290 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:57.844994 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:57.846427 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:57.846916 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:57.847011 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:57.850412 (+ 3401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:56:57.850416 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:56:57.850420 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:56:57.850450 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:56:57.853085 (+ 2635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:57.853719 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:57.853722 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:57.853725 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:57.853872 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:57.854228 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:57.854232 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:57.854234 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:57.854532 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:56:57.854754 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:57.855075 (+ 321us) server_negotiation.cc:299] Negotiation successful
0807 22:56:57.855321 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260807 22:56:57.857578 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 2451
I20260807 22:56:57.870291 3096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:57.858354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53270 (local address 127.25.94.1:46849)
0807 22:56:57.858519 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:57.858523 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:57.858938 (+ 415us) server_negotiation.cc:407] Connection header received
0807 22:56:57.859054 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:57.859057 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:57.859115 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:57.859193 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:56:57.860163 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:57.860875 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:57.862216 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:57.862702 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:57.862838 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:57.865064 (+ 2226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:56:57.865069 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:56:57.865073 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:56:57.865102 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:56:57.867820 (+ 2718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:57.868511 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:57.868515 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:57.868517 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:57.868657 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:57.868992 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:57.868995 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:57.868997 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:57.869372 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:56:57.869573 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:57.869860 (+ 287us) server_negotiation.cc:299] Negotiation successful
0807 22:56:57.870028 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260807 22:56:57.870923 2649 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 22:56:57.956776 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 2451
I20260807 22:56:57.979743 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 2665
Aug 07 22:56:57 dist-test-slave-z8x9 krb5kdc[2278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143417, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.2@KRBTEST.COM
I20260807 22:56:57.996531 3124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:57.980839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43064 (local address 127.25.94.2:38137)
0807 22:56:57.981556 (+ 717us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:57.981564 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:57.981591 (+ 27us) server_negotiation.cc:407] Connection header received
0807 22:56:57.981654 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:57.981661 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:57.981875 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:57.982010 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:56:57.983017 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:57.984096 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:57.985454 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:57.986069 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:57.986239 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:57.990464 (+ 4225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:56:57.990477 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:56:57.990492 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:56:57.990528 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:56:57.993901 (+ 3373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:57.994615 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:57.994623 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:57.994625 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:57.994766 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:57.995091 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:57.995094 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:57.995096 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:57.995440 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:56:57.995680 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:57.995943 (+ 263us) server_negotiation.cc:299] Negotiation successful
0807 22:56:57.996202 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":549,"thread_start_us":428,"threads_started":1}
I20260807 22:56:57.997967 2875 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 22:56:58.081455 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 2665
I20260807 22:56:58.105989 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 2888
Aug 07 22:56:58 dist-test-slave-z8x9 krb5kdc[2278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143417, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.3@KRBTEST.COM
I20260807 22:56:58.123580 3130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:56:58.107098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57082 (local address 127.25.94.3:42093)
0807 22:56:58.107681 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0807 22:56:58.107689 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 22:56:58.107727 (+ 38us) server_negotiation.cc:407] Connection header received
0807 22:56:58.107789 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:56:58.107795 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:56:58.108012 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:56:58.108186 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:56:58.109432 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:58.110564 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:58.111825 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:56:58.112384 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:56:58.112532 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:56:58.116977 (+ 4445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:56:58.116990 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:56:58.117006 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:56:58.117082 (+ 76us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:56:58.120182 (+ 3100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:58.121103 (+ 921us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:58.121112 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:58.121114 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:58.121366 (+ 252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:56:58.121886 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:56:58.121890 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:56:58.121893 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:56:58.122328 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:56:58.122836 (+ 508us) server_negotiation.cc:1035] Waiting for connection context
0807 22:56:58.123012 (+ 176us) server_negotiation.cc:299] Negotiation successful
0807 22:56:58.123236 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":262,"threads_started":1}
I20260807 22:56:58.124996 3077 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 22:56:58.211730 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 2888
I20260807 22:56:58.237128 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 2300
I20260807 22:56:58.238583 2403 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 22:56:58.346280 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 2300
2026-08-07T22:56:58Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (4466 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 22:56:58 dist-test-slave-z8x9 krb5kdc[3141](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 22:56:58 dist-test-slave-z8x9 krb5kdc[3141](info): set up 2 sockets
Aug 07 22:56:58 dist-test-slave-z8x9 krb5kdc[3141](info): commencing operation
krb5kdc: starting...
W20260807 22:56:59.693522 25976 mini_kdc.cc:121] Time spent starting KDC: real 1.297s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/krb5kdc/test-user.keytab.
Aug 07 22:56:59 dist-test-slave-z8x9 krb5kdc[3141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143419, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T22:56:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T22:56:59Z Disabled control of system clock
WARNING: no policy specified for oryx/127.25.94.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.25.94.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.25.94.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/krb5kdc/oryx_127.25.94.62.keytab.
Entry for principal oryx/127.25.94.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/krb5kdc/oryx_127.25.94.62.keytab.
I20260807 22:56:59.953943 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.62:32777
--webserver_interface=127.25.94.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/krb5kdc/oryx_127.25.94.62.keytab
--principal=oryx/127.25.94.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.20:33185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.94.62:32777
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:57:00.311515 3161 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:57:00.311866 3161 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:57:00.312014 3161 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:57:00.319623 3161 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 22:57:00.319765 3161 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:57:00.319855 3161 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:57:00.319942 3161 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 22:57:00.320052 3161 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 22:57:00.330816 3161 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:33185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.94.62:32777
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/krb5kdc/oryx_127.25.94.62.keytab
--principal=oryx/127.25.94.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.62:32777
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.94.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 22:57:00.332286 3161 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:57:00.333761 3161 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:57:00.346326 3166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:57:00.346232 3167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:57:00.346275 3169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:57:00.347397 3161 server_base.cc:1034] running on GCE node
I20260807 22:57:00.348282 3161 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:57:00.349758 3161 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:57:00.350973 3161 hybrid_clock.cc:648] HybridClock initialized: now 1786143420350933 us; error 26 us; skew 500 ppm
I20260807 22:57:00.351280 3161 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 22:57:00 dist-test-slave-z8x9 krb5kdc[3141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143420, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.94.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:57:00.359795 3161 init.cc:377] Logged in from keytab as oryx/127.25.94.62@KRBTEST.COM (short username oryx)
I20260807 22:57:00.364346 3161 webserver.cc:460] Webserver started at http://127.25.94.62:34357/ using document root <none> and password file <none>
I20260807 22:57:00.365568 3161 fs_manager.cc:359] Metadata directory not provided
I20260807 22:57:00.365731 3161 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:57:00.366137 3161 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:57:00.369901 3161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/master-0/data/instance:
uuid: "9828cb5c031840cfac64207adc69ca5b"
format_stamp: "Formatted at 2026-08-07 22:57:00 on dist-test-slave-z8x9"
server_key: "637e3cdc29a640a87ca7ecb223a47f26"
server_key_iv: "962be66d11015eb334b294a5035e3cc6"
server_key_version: "encryptionkey@0"
I20260807 22:57:00.370745 3161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/master-0/wal/instance:
uuid: "9828cb5c031840cfac64207adc69ca5b"
format_stamp: "Formatted at 2026-08-07 22:57:00 on dist-test-slave-z8x9"
server_key: "637e3cdc29a640a87ca7ecb223a47f26"
server_key_iv: "962be66d11015eb334b294a5035e3cc6"
server_key_version: "encryptionkey@0"
I20260807 22:57:00.376225 3161 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260807 22:57:00.379819 3179 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:57:00.381125 3161 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 22:57:00.381376 3161 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/master-0/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/master-0/wal
uuid: "9828cb5c031840cfac64207adc69ca5b"
format_stamp: "Formatted at 2026-08-07 22:57:00 on dist-test-slave-z8x9"
server_key: "637e3cdc29a640a87ca7ecb223a47f26"
server_key_iv: "962be66d11015eb334b294a5035e3cc6"
server_key_version: "encryptionkey@0"
I20260807 22:57:00.381582 3161 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:57:00.415357 3161 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:57:00.419981 3161 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:57:00.420282 3161 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:57:00.439126 3161 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.62:32777
I20260807 22:57:00.439159 3253 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.62:32777 every 8 connection(s)
I20260807 22:57:00.440567 3161 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
I20260807 22:57:00.444451 3256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:57:00.451115 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 3161
I20260807 22:57:00.451362 25976 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/master-0/wal/instance
I20260807 22:57:00.452237 25976 external_mini_cluster.cc:1428] Setting key 495416f6038c6a82568dc698098e550c
I20260807 22:57:00.456264 3256 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9828cb5c031840cfac64207adc69ca5b: Bootstrap starting.
I20260807 22:57:00.460103 3256 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9828cb5c031840cfac64207adc69ca5b: Neither blocks nor log segments found. Creating new log.
I20260807 22:57:00.461248 3256 log.cc:824] T 00000000000000000000000000000000 P 9828cb5c031840cfac64207adc69ca5b: Log is configured to *not* fsync() on all Append() calls
I20260807 22:57:00.464499 3256 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9828cb5c031840cfac64207adc69ca5b: No bootstrap required, opened a new log
Aug 07 22:57:00 dist-test-slave-z8x9 krb5kdc[3141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143419, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.25.94.62@KRBTEST.COM
I20260807 22:57:00.473115 3256 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9828cb5c031840cfac64207adc69ca5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9828cb5c031840cfac64207adc69ca5b" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 32777 } }
I20260807 22:57:00.473605 3256 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9828cb5c031840cfac64207adc69ca5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:57:00.473718 3256 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9828cb5c031840cfac64207adc69ca5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9828cb5c031840cfac64207adc69ca5b, State: Initialized, Role: FOLLOWER
I20260807 22:57:00.474421 3256 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9828cb5c031840cfac64207adc69ca5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9828cb5c031840cfac64207adc69ca5b" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 32777 } }
I20260807 22:57:00.474718 3256 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9828cb5c031840cfac64207adc69ca5b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 22:57:00.474872 3256 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9828cb5c031840cfac64207adc69ca5b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 22:57:00.475080 3256 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9828cb5c031840cfac64207adc69ca5b [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:57:00.477214 3262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:00.454610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45084 (local address 127.25.94.62:32777)
0807 22:57:00.455444 (+ 834us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:00.455456 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:00.455497 (+ 41us) server_negotiation.cc:407] Connection header received
0807 22:57:00.457331 (+ 1834us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:00.457353 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:00.457773 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:00.458107 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:00.460022 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:00.461544 (+ 1522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:00.463165 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:00.463848 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:00.463981 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:00.468871 (+ 4890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:57:00.468879 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:57:00.468895 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:57:00.468932 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:57:00.472441 (+ 3509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:00.473134 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:00.473142 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:00.473148 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:00.473376 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:00.473847 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:00.473852 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:00.473854 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:00.474579 (+ 725us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:57:00.474918 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0807 22:57:00.475331 (+ 413us) server_negotiation.cc:299] Negotiation successful
0807 22:57:00.475799 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":602,"thread_start_us":373,"threads_started":1}
I20260807 22:57:00.478538 3256 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9828cb5c031840cfac64207adc69ca5b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9828cb5c031840cfac64207adc69ca5b" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 32777 } }
I20260807 22:57:00.479115 3256 leader_election.cc:302] T 00000000000000000000000000000000 P 9828cb5c031840cfac64207adc69ca5b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9828cb5c031840cfac64207adc69ca5b; no voters:
I20260807 22:57:00.479781 3256 leader_election.cc:288] T 00000000000000000000000000000000 P 9828cb5c031840cfac64207adc69ca5b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 22:57:00.479934 3270 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9828cb5c031840cfac64207adc69ca5b [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:57:00.481081 3270 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9828cb5c031840cfac64207adc69ca5b [term 1 LEADER]: Becoming Leader. State: Replica: 9828cb5c031840cfac64207adc69ca5b, State: Running, Role: LEADER
I20260807 22:57:00.481951 3256 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9828cb5c031840cfac64207adc69ca5b [sys.catalog]: configured and running, proceeding with master startup.
I20260807 22:57:00.481864 3270 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9828cb5c031840cfac64207adc69ca5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9828cb5c031840cfac64207adc69ca5b" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 32777 } }
I20260807 22:57:00.484908 3273 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9828cb5c031840cfac64207adc69ca5b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9828cb5c031840cfac64207adc69ca5b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9828cb5c031840cfac64207adc69ca5b" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 32777 } } }
I20260807 22:57:00.485396 3273 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9828cb5c031840cfac64207adc69ca5b [sys.catalog]: This master's current role is: LEADER
I20260807 22:57:00.485280 3274 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9828cb5c031840cfac64207adc69ca5b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9828cb5c031840cfac64207adc69ca5b. Latest consensus state: current_term: 1 leader_uuid: "9828cb5c031840cfac64207adc69ca5b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9828cb5c031840cfac64207adc69ca5b" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 32777 } } }
I20260807 22:57:00.485592 3274 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9828cb5c031840cfac64207adc69ca5b [sys.catalog]: This master's current role is: LEADER
I20260807 22:57:00.488024 3285 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 22:57:00.492461 3285 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 22:57:00.501300 3285 catalog_manager.cc:1347] Generated new cluster ID: b39391c33b034542a865d23e1ca84d55
I20260807 22:57:00.501446 3285 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 22:57:00.513827 3285 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 22:57:00.515823 3285 catalog_manager.cc:1504] Loading token signing keys...
I20260807 22:57:00.526024 3285 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9828cb5c031840cfac64207adc69ca5b: Generated new TSK 0
I20260807 22:57:00.527266 3285 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.25.94.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.25.94.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.25.94.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/krb5kdc/oryx_127.25.94.1.keytab.
Entry for principal oryx/127.25.94.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/krb5kdc/oryx_127.25.94.1.keytab.
I20260807 22:57:00.585426 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.1:0
--local_ip_for_outbound_sockets=127.25.94.1
--webserver_interface=127.25.94.1
--webserver_port=0
--tserver_master_addrs=127.25.94.62:32777
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/krb5kdc/oryx_127.25.94.1.keytab
--principal=oryx/127.25.94.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.20:33185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:57:00.930958 3303 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:57:00.931255 3303 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:57:00.931445 3303 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:57:00.938906 3303 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:57:00.939025 3303 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:57:00.939201 3303 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.1
I20260807 22:57:00.948926 3303 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:33185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/krb5kdc/oryx_127.25.94.1.keytab
--principal=oryx/127.25.94.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.94.1
--webserver_port=0
--tserver_master_addrs=127.25.94.62:32777
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.1
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 22:57:00.950301 3303 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:57:00.951651 3303 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:57:00.965413 3312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:57:00.965435 3317 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:57:00.965360 3314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:57:00.966559 3303 server_base.cc:1034] running on GCE node
I20260807 22:57:00.967216 3303 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:57:00.968191 3303 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:57:00.969427 3303 hybrid_clock.cc:648] HybridClock initialized: now 1786143420969372 us; error 47 us; skew 500 ppm
I20260807 22:57:00.969753 3303 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 22:57:00 dist-test-slave-z8x9 krb5kdc[3141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143420, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.94.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:57:00.972951 3303 init.cc:377] Logged in from keytab as oryx/127.25.94.1@KRBTEST.COM (short username oryx)
I20260807 22:57:00.974587 3303 webserver.cc:460] Webserver started at http://127.25.94.1:36483/ using document root <none> and password file <none>
I20260807 22:57:00.975322 3303 fs_manager.cc:359] Metadata directory not provided
I20260807 22:57:00.975479 3303 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:57:00.975816 3303 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:57:00.979156 3303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/ts-0/data/instance:
uuid: "00fdb2f332f3437ea55e22ad5bf8ad14"
format_stamp: "Formatted at 2026-08-07 22:57:00 on dist-test-slave-z8x9"
server_key: "6155cdf0f39b74d11e287b90a9ef702a"
server_key_iv: "729f919dc147b53681cedae8f1d569d1"
server_key_version: "encryptionkey@0"
I20260807 22:57:00.980050 3303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/ts-0/wal/instance:
uuid: "00fdb2f332f3437ea55e22ad5bf8ad14"
format_stamp: "Formatted at 2026-08-07 22:57:00 on dist-test-slave-z8x9"
server_key: "6155cdf0f39b74d11e287b90a9ef702a"
server_key_iv: "729f919dc147b53681cedae8f1d569d1"
server_key_version: "encryptionkey@0"
I20260807 22:57:00.986555 3303 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260807 22:57:00.990280 3326 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:57:00.991665 3303 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260807 22:57:00.991858 3303 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/ts-0/wal
uuid: "00fdb2f332f3437ea55e22ad5bf8ad14"
format_stamp: "Formatted at 2026-08-07 22:57:00 on dist-test-slave-z8x9"
server_key: "6155cdf0f39b74d11e287b90a9ef702a"
server_key_iv: "729f919dc147b53681cedae8f1d569d1"
server_key_version: "encryptionkey@0"
I20260807 22:57:00.992079 3303 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:57:01.009548 3303 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:57:01.013872 3303 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:57:01.014189 3303 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:57:01.015231 3303 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:57:01.016703 3303 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:57:01.016814 3303 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:57:01.016991 3303 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:57:01.017068 3303 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:57:01.051195 3303 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.1:38459
I20260807 22:57:01.051257 3503 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.1:38459 every 8 connection(s)
I20260807 22:57:01.052573 3303 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/ts-0/data/info.pb
I20260807 22:57:01.062070 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 3303
I20260807 22:57:01.062314 25976 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/ts-0/wal/instance
I20260807 22:57:01.063146 25976 external_mini_cluster.cc:1428] Setting key 4b7fe7dad9b15efb340251ba83c55a00
Aug 07 22:57:01 dist-test-slave-z8x9 krb5kdc[3141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143420, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.94.1@KRBTEST.COM for oryx/127.25.94.62@KRBTEST.COM
I20260807 22:57:01.073607 3509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:01.055241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.1:51157 (local address 127.25.94.62:32777)
0807 22:57:01.055690 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:01.055695 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:01.056579 (+ 884us) server_negotiation.cc:407] Connection header received
0807 22:57:01.058465 (+ 1886us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:01.058471 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:01.058558 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:01.058676 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:01.060150 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:01.060898 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:01.063134 (+ 2236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:01.063595 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:01.063740 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:01.067639 (+ 3899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:57:01.067644 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:57:01.067648 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:57:01.067688 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:57:01.070453 (+ 2765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:01.071219 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:01.071229 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:01.071232 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:01.071349 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:01.071803 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:01.071808 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:01.071813 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:01.072079 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:57:01.072243 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0807 22:57:01.072938 (+ 695us) server_negotiation.cc:299] Negotiation successful
0807 22:57:01.073212 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":221,"threads_started":1}
I20260807 22:57:01.074589 3511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:01.055729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:32777 (local address 127.25.94.1:51157)
0807 22:57:01.056418 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:01.056456 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0807 22:57:01.058203 (+ 1747us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:57:01.058920 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:57:01.058928 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:57:01.059433 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:57:01.059970 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:01.059993 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:01.061069 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:01.061073 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:01.062966 (+ 1893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:01.062990 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:01.063831 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:01.063836 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:01.064027 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:01.064762 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:57:01.064799 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:57:01.067409 (+ 2610us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:57:01.070609 (+ 3200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:57:01.070618 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:57:01.070636 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:57:01.071075 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:57:01.071465 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:57:01.071470 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:57:01.071474 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:57:01.071674 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:57:01.072209 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:57:01.072217 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:57:01.072564 (+ 347us) client_negotiation.cc:769] Sending connection context
0807 22:57:01.072839 (+ 275us) client_negotiation.cc:240] Negotiation successful
0807 22:57:01.073421 (+ 582us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":270,"threads_started":1}
I20260807 22:57:01.076334 3504 heartbeater.cc:347] Connected to a master server at 127.25.94.62:32777
I20260807 22:57:01.076649 3504 heartbeater.cc:464] Registering TS with master...
I20260807 22:57:01.077533 3504 heartbeater.cc:511] Master 127.25.94.62:32777 requested a full tablet report, sending...
I20260807 22:57:01.079954 3206 ts_manager.cc:194] Registered new tserver with Master: 00fdb2f332f3437ea55e22ad5bf8ad14 (127.25.94.1:38459)
I20260807 22:57:01.082300 3206 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.25.94.1@KRBTEST.COM'} at 127.25.94.1:51157
WARNING: no policy specified for oryx/127.25.94.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.25.94.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.25.94.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/krb5kdc/oryx_127.25.94.2.keytab.
Entry for principal oryx/127.25.94.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/krb5kdc/oryx_127.25.94.2.keytab.
I20260807 22:57:01.122906 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.2:0
--local_ip_for_outbound_sockets=127.25.94.2
--webserver_interface=127.25.94.2
--webserver_port=0
--tserver_master_addrs=127.25.94.62:32777
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/krb5kdc/oryx_127.25.94.2.keytab
--principal=oryx/127.25.94.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.20:33185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:57:01.468055 3515 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:57:01.468326 3515 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:57:01.468420 3515 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:57:01.475764 3515 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:57:01.475876 3515 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:57:01.475988 3515 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.2
I20260807 22:57:01.486644 3515 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:33185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/krb5kdc/oryx_127.25.94.2.keytab
--principal=oryx/127.25.94.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.94.2
--webserver_port=0
--tserver_master_addrs=127.25.94.62:32777
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.2
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 22:57:01.487947 3515 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:57:01.489298 3515 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:57:01.502879 3526 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:57:01.502944 3530 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:57:01.502880 3527 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:57:01.503762 3515 server_base.cc:1034] running on GCE node
I20260807 22:57:01.504456 3515 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:57:01.505455 3515 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:57:01.506686 3515 hybrid_clock.cc:648] HybridClock initialized: now 1786143421506638 us; error 59 us; skew 500 ppm
I20260807 22:57:01.507004 3515 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 22:57:01 dist-test-slave-z8x9 krb5kdc[3141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143421, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.94.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:57:01.510281 3515 init.cc:377] Logged in from keytab as oryx/127.25.94.2@KRBTEST.COM (short username oryx)
I20260807 22:57:01.511857 3515 webserver.cc:460] Webserver started at http://127.25.94.2:39967/ using document root <none> and password file <none>
I20260807 22:57:01.512573 3515 fs_manager.cc:359] Metadata directory not provided
I20260807 22:57:01.512728 3515 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:57:01.513041 3515 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:57:01.516026 3515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/ts-1/data/instance:
uuid: "c9f865cfe1654d8f88a210773ef5d9bd"
format_stamp: "Formatted at 2026-08-07 22:57:01 on dist-test-slave-z8x9"
server_key: "ee33525b0ade321cc82eb098fda049e4"
server_key_iv: "78921ba40816d9ff0bc85e50e8ab8070"
server_key_version: "encryptionkey@0"
I20260807 22:57:01.516824 3515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/ts-1/wal/instance:
uuid: "c9f865cfe1654d8f88a210773ef5d9bd"
format_stamp: "Formatted at 2026-08-07 22:57:01 on dist-test-slave-z8x9"
server_key: "ee33525b0ade321cc82eb098fda049e4"
server_key_iv: "78921ba40816d9ff0bc85e50e8ab8070"
server_key_version: "encryptionkey@0"
I20260807 22:57:01.522186 3515 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260807 22:57:01.526182 3548 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:57:01.527642 3515 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260807 22:57:01.527824 3515 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/ts-1/wal
uuid: "c9f865cfe1654d8f88a210773ef5d9bd"
format_stamp: "Formatted at 2026-08-07 22:57:01 on dist-test-slave-z8x9"
server_key: "ee33525b0ade321cc82eb098fda049e4"
server_key_iv: "78921ba40816d9ff0bc85e50e8ab8070"
server_key_version: "encryptionkey@0"
I20260807 22:57:01.528005 3515 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:57:01.564880 3515 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:57:01.569298 3515 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:57:01.569598 3515 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:57:01.570583 3515 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:57:01.571945 3515 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:57:01.572050 3515 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:57:01.572201 3515 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:57:01.572266 3515 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:57:01.607848 3515 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.2:36449
I20260807 22:57:01.607932 3711 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.2:36449 every 8 connection(s)
I20260807 22:57:01.609151 3515 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/ts-1/data/info.pb
I20260807 22:57:01.619014 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 3515
I20260807 22:57:01.619202 25976 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/ts-1/wal/instance
I20260807 22:57:01.619683 25976 external_mini_cluster.cc:1428] Setting key c419787120f41836e2049ab2d78a63ce
Aug 07 22:57:01 dist-test-slave-z8x9 krb5kdc[3141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143421, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.94.2@KRBTEST.COM for oryx/127.25.94.62@KRBTEST.COM
I20260807 22:57:01.629825 3715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:01.611770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.2:39911 (local address 127.25.94.62:32777)
0807 22:57:01.612249 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:01.612253 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:01.613025 (+ 772us) server_negotiation.cc:407] Connection header received
0807 22:57:01.614856 (+ 1831us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:01.614864 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:01.614957 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:01.615076 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:01.616506 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:01.617328 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:01.618965 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:01.619505 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:01.619607 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:01.623696 (+ 4089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:57:01.623701 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:57:01.623705 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:57:01.623735 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:57:01.626724 (+ 2989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:01.627456 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:01.627463 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:01.627467 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:01.627610 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:01.628054 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:01.628059 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:01.628062 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:01.628340 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:57:01.628497 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0807 22:57:01.629204 (+ 707us) server_negotiation.cc:299] Negotiation successful
0807 22:57:01.629442 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":264,"threads_started":1}
I20260807 22:57:01.630777 3716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:01.612188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:32777 (local address 127.25.94.2:39911)
0807 22:57:01.612864 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:01.612907 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0807 22:57:01.614588 (+ 1681us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:57:01.615304 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:57:01.615311 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:57:01.615803 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:57:01.616320 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:01.616341 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:01.617489 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:01.617493 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:01.618821 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:01.618837 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:01.619723 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:01.619733 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:01.620049 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:01.620857 (+ 808us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:57:01.620887 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:57:01.623471 (+ 2584us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:57:01.626864 (+ 3393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:57:01.626873 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:57:01.626890 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:57:01.627333 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:57:01.627726 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:57:01.627731 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:57:01.627734 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:57:01.627931 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:57:01.628472 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:57:01.628480 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:57:01.628821 (+ 341us) client_negotiation.cc:769] Sending connection context
0807 22:57:01.629087 (+ 266us) client_negotiation.cc:240] Negotiation successful
0807 22:57:01.629660 (+ 573us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":272,"threads_started":1}
I20260807 22:57:01.632354 3713 heartbeater.cc:347] Connected to a master server at 127.25.94.62:32777
I20260807 22:57:01.632678 3713 heartbeater.cc:464] Registering TS with master...
I20260807 22:57:01.633602 3713 heartbeater.cc:511] Master 127.25.94.62:32777 requested a full tablet report, sending...
I20260807 22:57:01.635190 3206 ts_manager.cc:194] Registered new tserver with Master: c9f865cfe1654d8f88a210773ef5d9bd (127.25.94.2:36449)
I20260807 22:57:01.636006 3206 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.25.94.2@KRBTEST.COM'} at 127.25.94.2:39911
WARNING: no policy specified for oryx/127.25.94.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.25.94.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.25.94.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/krb5kdc/oryx_127.25.94.3.keytab.
Entry for principal oryx/127.25.94.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/krb5kdc/oryx_127.25.94.3.keytab.
I20260807 22:57:01.677241 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.3:0
--local_ip_for_outbound_sockets=127.25.94.3
--webserver_interface=127.25.94.3
--webserver_port=0
--tserver_master_addrs=127.25.94.62:32777
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/krb5kdc/oryx_127.25.94.3.keytab
--principal=oryx/127.25.94.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.20:33185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:57:02.015347 3719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:57:02.015666 3719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:57:02.015779 3719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:57:02.023838 3719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:57:02.023972 3719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:57:02.024089 3719 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.3
I20260807 22:57:02.034416 3719 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:33185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/krb5kdc/oryx_127.25.94.3.keytab
--principal=oryx/127.25.94.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.94.3
--webserver_port=0
--tserver_master_addrs=127.25.94.62:32777
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.3
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 22:57:02.035727 3719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:57:02.037057 3719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:57:02.051059 3731 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:57:02.051153 3734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:57:02.051088 3730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:57:02.052734 3719 server_base.cc:1034] running on GCE node
I20260807 22:57:02.053524 3719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:57:02.054821 3719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:57:02.056082 3719 hybrid_clock.cc:648] HybridClock initialized: now 1786143422055991 us; error 81 us; skew 500 ppm
I20260807 22:57:02.056457 3719 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 22:57:02 dist-test-slave-z8x9 krb5kdc[3141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143422, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.94.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:57:02.059937 3719 init.cc:377] Logged in from keytab as oryx/127.25.94.3@KRBTEST.COM (short username oryx)
I20260807 22:57:02.061479 3719 webserver.cc:460] Webserver started at http://127.25.94.3:40397/ using document root <none> and password file <none>
I20260807 22:57:02.062160 3719 fs_manager.cc:359] Metadata directory not provided
I20260807 22:57:02.062274 3719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:57:02.062515 3719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:57:02.065580 3719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/ts-2/data/instance:
uuid: "974ee5dfbcb449f8849a6e84a6fff395"
format_stamp: "Formatted at 2026-08-07 22:57:02 on dist-test-slave-z8x9"
server_key: "39bff9f91deddba698c9f750898a6aea"
server_key_iv: "49d298e22dd350608f3df02a620bc48d"
server_key_version: "encryptionkey@0"
I20260807 22:57:02.066347 3719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/ts-2/wal/instance:
uuid: "974ee5dfbcb449f8849a6e84a6fff395"
format_stamp: "Formatted at 2026-08-07 22:57:02 on dist-test-slave-z8x9"
server_key: "39bff9f91deddba698c9f750898a6aea"
server_key_iv: "49d298e22dd350608f3df02a620bc48d"
server_key_version: "encryptionkey@0"
I20260807 22:57:02.071635 3719 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.002s
I20260807 22:57:02.075150 3743 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:57:02.076563 3719 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 22:57:02.076843 3719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/ts-2/wal
uuid: "974ee5dfbcb449f8849a6e84a6fff395"
format_stamp: "Formatted at 2026-08-07 22:57:02 on dist-test-slave-z8x9"
server_key: "39bff9f91deddba698c9f750898a6aea"
server_key_iv: "49d298e22dd350608f3df02a620bc48d"
server_key_version: "encryptionkey@0"
I20260807 22:57:02.077049 3719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:57:02.086033 3504 heartbeater.cc:503] Master 127.25.94.62:32777 was elected leader, sending a full tablet report...
I20260807 22:57:02.090718 3719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:57:02.094990 3719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:57:02.095325 3719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:57:02.096443 3719 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:57:02.098166 3719 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:57:02.098261 3719 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:57:02.098393 3719 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:57:02.098443 3719 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:57:02.134750 3719 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.3:37625
I20260807 22:57:02.134801 3917 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.3:37625 every 8 connection(s)
I20260807 22:57:02.136098 3719 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/ts-2/data/info.pb
I20260807 22:57:02.142204 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 3719
I20260807 22:57:02.142419 25976 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/ts-2/wal/instance
I20260807 22:57:02.142982 25976 external_mini_cluster.cc:1428] Setting key 1395d3d337c7f18cb2e3dd7aa3a040c0
Aug 07 22:57:02 dist-test-slave-z8x9 krb5kdc[3141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143422, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.94.3@KRBTEST.COM for oryx/127.25.94.62@KRBTEST.COM
I20260807 22:57:02.159109 3923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:02.139170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.3:50223 (local address 127.25.94.62:32777)
0807 22:57:02.139672 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:02.139677 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:02.140719 (+ 1042us) server_negotiation.cc:407] Connection header received
0807 22:57:02.143152 (+ 2433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:02.143159 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:02.143262 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:02.143366 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:02.145453 (+ 2087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:02.146268 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:02.147895 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:02.148304 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:02.148440 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:02.152270 (+ 3830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:57:02.152278 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:57:02.152283 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:57:02.152331 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:57:02.155530 (+ 3199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:02.156456 (+ 926us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:02.156466 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:02.156468 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:02.156604 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:02.157089 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:02.157093 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:02.157095 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:02.157471 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:57:02.157647 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0807 22:57:02.158415 (+ 768us) server_negotiation.cc:299] Negotiation successful
0807 22:57:02.158689 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":227,"threads_started":1}
I20260807 22:57:02.160174 3924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:02.139769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:32777 (local address 127.25.94.3:50223)
0807 22:57:02.140528 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:02.140569 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0807 22:57:02.142745 (+ 2176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:57:02.143689 (+ 944us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:57:02.143698 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:57:02.144513 (+ 815us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:57:02.145252 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:02.145273 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:02.146450 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:02.146456 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:02.147736 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:02.147749 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:02.148563 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:02.148570 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:02.148822 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:02.149608 (+ 786us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:57:02.149637 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:57:02.151987 (+ 2350us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:57:02.155735 (+ 3748us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:57:02.155747 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:57:02.155769 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:57:02.156304 (+ 535us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:57:02.156723 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:57:02.156728 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:57:02.156732 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:57:02.156950 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:57:02.157611 (+ 661us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:57:02.157620 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:57:02.158006 (+ 386us) client_negotiation.cc:769] Sending connection context
0807 22:57:02.158318 (+ 312us) client_negotiation.cc:240] Negotiation successful
0807 22:57:02.158910 (+ 592us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":319,"threads_started":1}
I20260807 22:57:02.161818 3920 heartbeater.cc:347] Connected to a master server at 127.25.94.62:32777
I20260807 22:57:02.162217 3920 heartbeater.cc:464] Registering TS with master...
I20260807 22:57:02.163071 3920 heartbeater.cc:511] Master 127.25.94.62:32777 requested a full tablet report, sending...
I20260807 22:57:02.164717 3206 ts_manager.cc:194] Registered new tserver with Master: 974ee5dfbcb449f8849a6e84a6fff395 (127.25.94.3:37625)
I20260807 22:57:02.165652 3206 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.25.94.3@KRBTEST.COM'} at 127.25.94.3:50223
I20260807 22:57:02.171574 25976 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.25.94.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.25.94.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.25.94.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/krb5kdc/oryx_127.25.94.61.keytab.
Entry for principal oryx/127.25.94.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/krb5kdc/oryx_127.25.94.61.keytab.
I20260807 22:57:02.234318 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.61:45205
--webserver_interface=127.25.94.61
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/krb5kdc/oryx_127.25.94.61.keytab
--principal=oryx/127.25.94.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.20:33185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.94.62:32777,127.25.94.61:45205
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:57:02.587661 3930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:57:02.587983 3930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:57:02.588078 3930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:57:02.595685 3930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 22:57:02.595811 3930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:57:02.595927 3930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:57:02.596040 3930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 22:57:02.596105 3930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 22:57:02.607206 3930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:33185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.25.94.62:32777,127.25.94.61:45205
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/krb5kdc/oryx_127.25.94.61.keytab
--principal=oryx/127.25.94.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.61:45205
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.25.94.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
W20260807 22:57:02.607662 3930 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.25.94.62:32777,127.25.94.61:45205'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260807 22:57:02.608754 3930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:57:02.610137 3930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:57:02.622496 3947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:57:02.622584 3944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:57:02.622594 3943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:57:02.626732 3930 server_base.cc:1034] running on GCE node
I20260807 22:57:02.627663 3930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:57:02.629150 3930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:57:02.630560 3930 hybrid_clock.cc:648] HybridClock initialized: now 1786143422630498 us; error 33 us; skew 500 ppm
I20260807 22:57:02.630947 3930 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 22:57:02 dist-test-slave-z8x9 krb5kdc[3141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143422, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.94.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:57:02.634686 3930 init.cc:377] Logged in from keytab as oryx/127.25.94.61@KRBTEST.COM (short username oryx)
I20260807 22:57:02.636368 3930 webserver.cc:460] Webserver started at http://127.25.94.61:35335/ using document root <none> and password file <none>
I20260807 22:57:02.637142 3930 fs_manager.cc:359] Metadata directory not provided
I20260807 22:57:02.637355 3930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:57:02.637689 3930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:57:02.639257 3713 heartbeater.cc:503] Master 127.25.94.62:32777 was elected leader, sending a full tablet report...
I20260807 22:57:02.641109 3930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/master-1/data/instance:
uuid: "5afdd40906094502ab5d825fbe1315a7"
format_stamp: "Formatted at 2026-08-07 22:57:02 on dist-test-slave-z8x9"
server_key: "36e3a87d77982aa803a3d0806a32d898"
server_key_iv: "8d3d5c7c6213dfb713b52a7a495e706d"
server_key_version: "encryptionkey@0"
I20260807 22:57:02.641971 3930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/master-1/wal/instance:
uuid: "5afdd40906094502ab5d825fbe1315a7"
format_stamp: "Formatted at 2026-08-07 22:57:02 on dist-test-slave-z8x9"
server_key: "36e3a87d77982aa803a3d0806a32d898"
server_key_iv: "8d3d5c7c6213dfb713b52a7a495e706d"
server_key_version: "encryptionkey@0"
I20260807 22:57:02.647651 3930 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260807 22:57:02.651135 3963 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:57:02.652531 3930 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 22:57:02.652737 3930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/master-1/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/master-1/wal
uuid: "5afdd40906094502ab5d825fbe1315a7"
format_stamp: "Formatted at 2026-08-07 22:57:02 on dist-test-slave-z8x9"
server_key: "36e3a87d77982aa803a3d0806a32d898"
server_key_iv: "8d3d5c7c6213dfb713b52a7a495e706d"
server_key_version: "encryptionkey@0"
I20260807 22:57:02.652972 3930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:57:02.674913 3930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:57:02.679555 3930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:57:02.679827 3930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:57:02.700017 3930 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.61:45205
I20260807 22:57:02.700037 4042 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.61:45205 every 8 connection(s)
I20260807 22:57:02.701489 3930 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/master-1/data/info.pb
I20260807 22:57:02.705282 4046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:57:02.709944 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 3930
I20260807 22:57:02.712956 4046 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.94.62" port: 32777 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 22:57:02 dist-test-slave-z8x9 krb5kdc[3141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143419, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.25.94.61@KRBTEST.COM
Aug 07 22:57:02 dist-test-slave-z8x9 krb5kdc[3141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143422, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.94.61@KRBTEST.COM for oryx/127.25.94.62@KRBTEST.COM
I20260807 22:57:02.734016 4052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:02.717072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41300 (local address 127.25.94.62:32777)
0807 22:57:02.718350 (+ 1278us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:02.718355 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:02.718369 (+ 14us) server_negotiation.cc:407] Connection header received
0807 22:57:02.718419 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:02.718422 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:02.718495 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:02.718624 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:02.719784 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:02.720496 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:02.722024 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:02.722742 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:02.722874 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:02.726465 (+ 3591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:57:02.726474 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:57:02.726478 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:57:02.726525 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:57:02.730290 (+ 3765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:02.731254 (+ 964us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:02.731260 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:02.731263 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:02.731465 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:02.731990 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:02.731994 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:02.731998 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:02.732492 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:57:02.732659 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0807 22:57:02.733415 (+ 756us) server_negotiation.cc:299] Negotiation successful
0807 22:57:02.733654 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1179,"thread_start_us":1093,"threads_started":1}
I20260807 22:57:02.735015 4050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:02.712384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57796 (local address 127.25.94.61:45205)
0807 22:57:02.713304 (+ 920us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:02.713315 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:02.713358 (+ 43us) server_negotiation.cc:407] Connection header received
0807 22:57:02.714970 (+ 1612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:02.714993 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:02.715425 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:02.715727 (+ 302us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:02.717742 (+ 2015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:02.719285 (+ 1543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:02.720535 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:02.721260 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:02.721443 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:02.726023 (+ 4580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:57:02.726031 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:57:02.726044 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:57:02.726074 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:57:02.729999 (+ 3925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:02.730988 (+ 989us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:02.730995 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:02.730998 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:02.731277 (+ 279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:02.731830 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:02.731835 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:02.731838 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:02.732586 (+ 748us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:57:02.732903 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0807 22:57:02.733323 (+ 420us) server_negotiation.cc:299] Negotiation successful
0807 22:57:02.733799 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":643,"thread_start_us":431,"threads_started":1}
I20260807 22:57:02.735034 4051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:02.717025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:32777 (local address 127.0.0.1:41300)
0807 22:57:02.717794 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:02.717835 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0807 22:57:02.717981 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:57:02.718687 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:57:02.718694 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:57:02.719350 (+ 656us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:57:02.719613 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:02.719621 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:02.720631 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:02.720639 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:02.721833 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:02.721846 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:02.722965 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:02.722969 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:02.723096 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:02.723714 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:57:02.723737 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:57:02.726261 (+ 2524us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:57:02.730442 (+ 4181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:57:02.730455 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:57:02.730478 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:57:02.731117 (+ 639us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:57:02.731604 (+ 487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:57:02.731608 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:57:02.731612 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:57:02.731869 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:57:02.732630 (+ 761us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:57:02.732636 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:57:02.732920 (+ 284us) client_negotiation.cc:769] Sending connection context
0807 22:57:02.733323 (+ 403us) client_negotiation.cc:240] Negotiation successful
0807 22:57:02.733958 (+ 635us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"mutex_wait_us":122,"thread_start_us":388,"threads_started":1}
I20260807 22:57:02.737298 4046 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.94.61" port: 45205 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 22:57:02 dist-test-slave-z8x9 krb5kdc[3141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143422, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.94.61@KRBTEST.COM for oryx/127.25.94.61@KRBTEST.COM
I20260807 22:57:02.750823 4051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:02.738234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.61:45205 (local address 127.0.0.1:57810)
0807 22:57:02.738439 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:02.738459 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 22:57:02.738560 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:57:02.738926 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:57:02.738930 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:57:02.739341 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:57:02.739476 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:02.739484 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:02.740452 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:02.740458 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:02.741399 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:02.741411 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:02.742090 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:02.742095 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:02.742347 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:02.742989 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:57:02.743006 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:57:02.745250 (+ 2244us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:57:02.748199 (+ 2949us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:57:02.748206 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:57:02.748210 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:57:02.748614 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:57:02.748994 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:57:02.749001 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:57:02.749005 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:57:02.749141 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:57:02.749786 (+ 645us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:57:02.749790 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:57:02.749948 (+ 158us) client_negotiation.cc:769] Sending connection context
0807 22:57:02.750088 (+ 140us) client_negotiation.cc:240] Negotiation successful
0807 22:57:02.750413 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54,"mutex_wait_us":1}
I20260807 22:57:02.750823 4050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:02.738105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57810 (local address 127.25.94.61:45205)
0807 22:57:02.738701 (+ 596us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:02.738705 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:02.738718 (+ 13us) server_negotiation.cc:407] Connection header received
0807 22:57:02.738750 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:02.738753 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:02.738808 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:02.738911 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:02.739610 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:02.740310 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:02.741541 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:02.741942 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:02.742081 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:02.745480 (+ 3399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:57:02.745485 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:57:02.745488 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:57:02.745519 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:57:02.748052 (+ 2533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:02.748746 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:02.748753 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:02.748761 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:02.748886 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:02.749359 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:02.749366 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:02.749369 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:02.749668 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:57:02.749826 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0807 22:57:02.750184 (+ 358us) server_negotiation.cc:299] Negotiation successful
0807 22:57:02.750542 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":207,"server-negotiator.queue_time_us":103}
I20260807 22:57:02.755980 4046 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5afdd40906094502ab5d825fbe1315a7: Bootstrap starting.
I20260807 22:57:02.759238 4046 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5afdd40906094502ab5d825fbe1315a7: Neither blocks nor log segments found. Creating new log.
I20260807 22:57:02.760336 4046 log.cc:824] T 00000000000000000000000000000000 P 5afdd40906094502ab5d825fbe1315a7: Log is configured to *not* fsync() on all Append() calls
I20260807 22:57:02.763782 4046 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5afdd40906094502ab5d825fbe1315a7: No bootstrap required, opened a new log
I20260807 22:57:02.771482 4046 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5afdd40906094502ab5d825fbe1315a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9828cb5c031840cfac64207adc69ca5b" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 32777 } } peers { permanent_uuid: "5afdd40906094502ab5d825fbe1315a7" member_type: VOTER last_known_addr { host: "127.25.94.61" port: 45205 } }
I20260807 22:57:02.771898 4046 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5afdd40906094502ab5d825fbe1315a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:57:02.772020 4046 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5afdd40906094502ab5d825fbe1315a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5afdd40906094502ab5d825fbe1315a7, State: Initialized, Role: FOLLOWER
I20260807 22:57:02.772632 4046 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5afdd40906094502ab5d825fbe1315a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9828cb5c031840cfac64207adc69ca5b" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 32777 } } peers { permanent_uuid: "5afdd40906094502ab5d825fbe1315a7" member_type: VOTER last_known_addr { host: "127.25.94.61" port: 45205 } }
I20260807 22:57:02.773650 4054 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5afdd40906094502ab5d825fbe1315a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9828cb5c031840cfac64207adc69ca5b" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 32777 } } peers { permanent_uuid: "5afdd40906094502ab5d825fbe1315a7" member_type: VOTER last_known_addr { host: "127.25.94.61" port: 45205 } } }
I20260807 22:57:02.773964 4054 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5afdd40906094502ab5d825fbe1315a7 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 22:57:02.774698 4046 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5afdd40906094502ab5d825fbe1315a7 [sys.catalog]: configured and running, proceeding with master startup.
W20260807 22:57:02.784475 4070 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5afdd40906094502ab5d825fbe1315a7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 22:57:02.784657 4070 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 22:57:02.788199 3930 master_runner.cc:428] Detected that this master 5afdd40906094502ab5d825fbe1315a7 is joining an existing cluster
I20260807 22:57:02.788322 3930 master_runner.cc:432] Initiating AddMaster RPC to add 127.25.94.61:45205
I20260807 22:57:02.818459 4052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:02.793473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41314 (local address 127.25.94.62:32777)
0807 22:57:02.793666 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:02.793672 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:02.803222 (+ 9550us) server_negotiation.cc:407] Connection header received
0807 22:57:02.803493 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:02.803500 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:02.803604 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:02.803679 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:02.805451 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:02.806503 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:02.808090 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:02.808594 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:02.808719 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:02.811247 (+ 2528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:57:02.811254 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:57:02.811258 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:57:02.811298 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:57:02.815015 (+ 3717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:02.815867 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:02.815873 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:02.815876 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:02.816060 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:02.816514 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:02.816518 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:02.816521 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:02.816873 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:57:02.816993 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0807 22:57:02.817639 (+ 646us) server_negotiation.cc:299] Negotiation successful
0807 22:57:02.818058 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260807 22:57:02.819725 4083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:02.794127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:32777 (local address 127.0.0.1:41314)
0807 22:57:02.803077 (+ 8950us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:02.803105 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0807 22:57:02.803334 (+ 229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:57:02.803916 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:57:02.803921 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:57:02.804741 (+ 820us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:57:02.805204 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:02.805225 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:02.806660 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:02.806665 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:02.807933 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:02.807946 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:02.808803 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:02.808810 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:02.809033 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:02.809899 (+ 866us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:57:02.809922 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:57:02.811038 (+ 1116us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:57:02.815169 (+ 4131us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:57:02.815174 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:57:02.815178 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:57:02.815715 (+ 537us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:57:02.816214 (+ 499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:57:02.816218 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:57:02.816221 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:57:02.816396 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:57:02.817069 (+ 673us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:57:02.817074 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:57:02.817340 (+ 266us) client_negotiation.cc:769] Sending connection context
0807 22:57:02.818747 (+ 1407us) client_negotiation.cc:240] Negotiation successful
0807 22:57:02.819119 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8817,"mutex_wait_us":65,"thread_start_us":8099,"threads_started":1}
I20260807 22:57:02.820118 4050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:02.793445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57826 (local address 127.25.94.61:45205)
0807 22:57:02.793640 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:02.793648 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:02.802825 (+ 9177us) server_negotiation.cc:407] Connection header received
0807 22:57:02.803069 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:02.803076 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:02.803156 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:02.803249 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:02.804993 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:02.806346 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:02.808542 (+ 2196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:02.809227 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:02.809398 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:02.811284 (+ 1886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:57:02.811290 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:57:02.811295 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:57:02.811326 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:57:02.815039 (+ 3713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:02.817023 (+ 1984us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:02.817028 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:02.817032 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:02.817228 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:02.817671 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:02.817677 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:02.817680 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:02.818092 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:57:02.818223 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0807 22:57:02.819240 (+ 1017us) server_negotiation.cc:299] Negotiation successful
0807 22:57:02.819491 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":60}
I20260807 22:57:02.821426 4085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:02.794064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.61:45205 (local address 127.0.0.1:57826)
0807 22:57:02.802673 (+ 8609us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:02.802706 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0807 22:57:02.802951 (+ 245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:57:02.803431 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:57:02.803437 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:57:02.804333 (+ 896us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:57:02.804813 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:02.804839 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:02.806553 (+ 1714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:02.806559 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:02.808388 (+ 1829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:02.808403 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:02.809436 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:02.809442 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:02.809634 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:02.810235 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:57:02.810251 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:57:02.811041 (+ 790us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:57:02.815192 (+ 4151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:57:02.815197 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:57:02.815201 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:57:02.816894 (+ 1693us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:57:02.817377 (+ 483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:57:02.817382 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:57:02.817385 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:57:02.817552 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:57:02.818306 (+ 754us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:57:02.818310 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:57:02.818931 (+ 621us) client_negotiation.cc:769] Sending connection context
0807 22:57:02.820628 (+ 1697us) client_negotiation.cc:240] Negotiation successful
0807 22:57:02.820915 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8374,"mutex_wait_us":31,"thread_start_us":8662,"threads_started":1}
WARNING: no policy specified for oryx/127.25.94.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.25.94.60@KRBTEST.COM" created.
Aug 07 22:57:02 dist-test-slave-z8x9 krb5kdc[3141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143420, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.94.62@KRBTEST.COM for oryx/127.25.94.61@KRBTEST.COM
I20260807 22:57:02.843359 4050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:02.827316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57828 (local address 127.25.94.61:45205)
0807 22:57:02.827538 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:02.827544 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:02.828355 (+ 811us) server_negotiation.cc:407] Connection header received
0807 22:57:02.828716 (+ 361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:02.828723 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:02.828859 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:02.828988 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:02.830151 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:02.830853 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:02.832293 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:02.832857 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:02.833008 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:02.836333 (+ 3325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:57:02.836339 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:57:02.836342 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:57:02.836372 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:57:02.839868 (+ 3496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:02.840812 (+ 944us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:02.840818 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:02.840822 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:02.841001 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:02.841515 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:02.841521 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:02.841525 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:02.841919 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:57:02.842071 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0807 22:57:02.842716 (+ 645us) server_negotiation.cc:299] Negotiation successful
0807 22:57:02.842989 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260807 22:57:02.843580 4089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:02.827090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.61:45205 (local address 127.0.0.1:57828)
0807 22:57:02.828192 (+ 1102us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:02.828239 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0807 22:57:02.828490 (+ 251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:57:02.829034 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:57:02.829042 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:57:02.829760 (+ 718us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:57:02.829985 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:02.829995 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:02.830988 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:02.830994 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:02.832120 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:02.832154 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:02.833030 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:02.833036 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:02.833279 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:02.833951 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:57:02.833968 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:57:02.836130 (+ 2162us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:57:02.840007 (+ 3877us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:57:02.840025 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:57:02.840036 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:57:02.840669 (+ 633us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:57:02.841136 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:57:02.841143 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:57:02.841150 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:57:02.841391 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:57:02.842068 (+ 677us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:57:02.842079 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:57:02.842332 (+ 253us) client_negotiation.cc:769] Sending connection context
0807 22:57:02.842572 (+ 240us) client_negotiation.cc:240] Negotiation successful
0807 22:57:02.842964 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":763,"thread_start_us":558,"threads_started":1}
I20260807 22:57:02.846231 3206 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.25.94.61:45205: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "5afdd40906094502ab5d825fbe1315a7"
member_type: NON_VOTER
last_known_addr {
host: "127.25.94.61"
port: 45205
}
attrs {
promote: true
}
}
dest_uuid: "9828cb5c031840cfac64207adc69ca5b"
cas_config_opid_index: -1
I20260807 22:57:02.847851 3206 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9828cb5c031840cfac64207adc69ca5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9828cb5c031840cfac64207adc69ca5b" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 32777 } } peers { permanent_uuid: "5afdd40906094502ab5d825fbe1315a7" member_type: NON_VOTER last_known_addr { host: "127.25.94.61" port: 45205 } attrs { promote: true } }
I20260807 22:57:02.856346 4000 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5afdd40906094502ab5d825fbe1315a7 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:57:02.855980 4098 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9828cb5c031840cfac64207adc69ca5b [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 5afdd40906094502ab5d825fbe1315a7 (127.25.94.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9828cb5c031840cfac64207adc69ca5b" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 32777 } } peers { permanent_uuid: "5afdd40906094502ab5d825fbe1315a7" member_type: NON_VOTER last_known_addr { host: "127.25.94.61" port: 45205 } attrs { promote: true } } }
I20260807 22:57:02.860860 4098 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.25.94.61:45205
I20260807 22:57:02.861326 4000 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5afdd40906094502ab5d825fbe1315a7 [term 1 FOLLOWER]: Refusing update from remote peer 9828cb5c031840cfac64207adc69ca5b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260807 22:57:02.861908 3930 master.cc:537] Master@127.25.94.61:45205 shutting down...
I20260807 22:57:02.862571 4098 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9828cb5c031840cfac64207adc69ca5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5afdd40906094502ab5d825fbe1315a7" member_type: NON_VOTER last_known_addr { host: "127.25.94.61" port: 45205 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 22:57:02.862978 4097 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9828cb5c031840cfac64207adc69ca5b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "9828cb5c031840cfac64207adc69ca5b" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9828cb5c031840cfac64207adc69ca5b" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 32777 } } peers { permanent_uuid: "5afdd40906094502ab5d825fbe1315a7" member_type: NON_VOTER last_known_addr { host: "127.25.94.61" port: 45205 } attrs { promote: true } } }
I20260807 22:57:02.863335 4097 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9828cb5c031840cfac64207adc69ca5b [sys.catalog]: This master's current role is: LEADER
W20260807 22:57:02.867218 3183 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.25.94.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/krb5kdc/oryx_127.25.94.60.keytab.
Entry for principal oryx/127.25.94.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/krb5kdc/oryx_127.25.94.60.keytab.
I20260807 22:57:02.871869 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.60:40149
--webserver_interface=127.25.94.60
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/krb5kdc/oryx_127.25.94.60.keytab
--principal=oryx/127.25.94.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.20:33185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.94.62:32777,127.25.94.61:45205,127.25.94.60:40149
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 22:57:02.885017 3930 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 5afdd40906094502ab5d825fbe1315a7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 22:57:02.885648 3930 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 5afdd40906094502ab5d825fbe1315a7 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260807 22:57:02.885581 3183 consensus_peers.cc:487] T 00000000000000000000000000000000 P 9828cb5c031840cfac64207adc69ca5b -> Peer 5afdd40906094502ab5d825fbe1315a7 (127.25.94.61:45205): Couldn't send request to peer 5afdd40906094502ab5d825fbe1315a7. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260807 22:57:02.886003 3930 tablet_replica.cc:330] T 00000000000000000000000000000000 P 5afdd40906094502ab5d825fbe1315a7: stopping tablet replica
I20260807 22:57:02.893573 3930 master.cc:552] Master@127.25.94.61:45205 shutdown complete.
I20260807 22:57:02.893731 3930 master_runner.cc:305] Clearing existing system tablet
I20260807 22:57:02.894212 3930 fs_manager.cc:359] Metadata directory not provided
I20260807 22:57:02.894361 3930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:57:02.897545 3930 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260807 22:57:02.899304 4108 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:57:02.900156 3930 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:57:02.900441 3930 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260807 22:57:02.900528 3930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/master-1/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/master-1/wal
uuid: "5afdd40906094502ab5d825fbe1315a7"
format_stamp: "Formatted at 2026-08-07 22:57:02 on dist-test-slave-z8x9"
server_key: "36e3a87d77982aa803a3d0806a32d898"
server_key_iv: "8d3d5c7c6213dfb713b52a7a495e706d"
server_key_version: "encryptionkey@0"
I20260807 22:57:02.906162 3930 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 5afdd40906094502ab5d825fbe1315a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 22:57:02.913391 3930 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 5afdd40906094502ab5d825fbe1315a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260807 22:57:02.913599 3930 log.cc:1196] T 00000000000000000000000000000000 P 5afdd40906094502ab5d825fbe1315a7: Deleting WAL directory at /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260807 22:57:02.914093 3930 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 5afdd40906094502ab5d825fbe1315a7: Deleting consensus metadata
I20260807 22:57:02.914377 3930 master_runner.cc:315] Copying system tablet from 127.25.94.62:32777
I20260807 22:57:02.917392 3930 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 5afdd40906094502ab5d825fbe1315a7: tablet copy: Beginning tablet copy session from remote peer at address 127.25.94.62:32777
I20260807 22:57:02.937139 4052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:02.918537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41326 (local address 127.25.94.62:32777)
0807 22:57:02.918736 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:02.918742 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:02.921855 (+ 3113us) server_negotiation.cc:407] Connection header received
0807 22:57:02.922107 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:02.922113 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:02.922218 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:02.922347 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:02.923506 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:02.924657 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:02.926548 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:02.927205 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:02.927350 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:02.929783 (+ 2433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:57:02.929792 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:57:02.929799 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:57:02.929843 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:57:02.933767 (+ 3924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:02.934716 (+ 949us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:02.934722 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:02.934727 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:02.934906 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:02.935403 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:02.935408 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:02.935413 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:02.935834 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:57:02.935997 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0807 22:57:02.936496 (+ 499us) server_negotiation.cc:299] Negotiation successful
0807 22:57:02.936760 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260807 22:57:02.937161 4117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:02.918464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:32777 (local address 127.0.0.1:41326)
0807 22:57:02.921595 (+ 3131us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:02.921621 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0807 22:57:02.921951 (+ 330us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:57:02.922392 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:57:02.922397 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:57:02.923012 (+ 615us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:57:02.923316 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:02.923336 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:02.924829 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:02.924835 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:02.926377 (+ 1542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:02.926394 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:02.927378 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:02.927385 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:02.927576 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:02.928357 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:57:02.928381 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:57:02.929533 (+ 1152us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:57:02.933938 (+ 4405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:57:02.933944 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:57:02.933948 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:57:02.934546 (+ 598us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:57:02.935038 (+ 492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:57:02.935043 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:57:02.935049 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:57:02.935255 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:57:02.935982 (+ 727us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:57:02.935987 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:57:02.936184 (+ 197us) client_negotiation.cc:769] Sending connection context
0807 22:57:02.936353 (+ 169us) client_negotiation.cc:240] Negotiation successful
0807 22:57:02.936677 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2976,"thread_start_us":2869,"threads_started":1}
I20260807 22:57:02.938613 3235 tablet_copy_service.cc:140] P 9828cb5c031840cfac64207adc69ca5b: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 5afdd40906094502ab5d825fbe1315a7 ({username='oryx', principal='oryx/127.25.94.61@KRBTEST.COM'} at 127.0.0.1:41326)
I20260807 22:57:02.939118 3235 tablet_copy_service.cc:161] P 9828cb5c031840cfac64207adc69ca5b: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 5afdd40906094502ab5d825fbe1315a7 at {username='oryx', principal='oryx/127.25.94.61@KRBTEST.COM'} at 127.0.0.1:41326: session id = 5afdd40906094502ab5d825fbe1315a7-00000000000000000000000000000000
I20260807 22:57:02.944789 3235 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 9828cb5c031840cfac64207adc69ca5b: Tablet Copy: opened 0 blocks and 1 log segments
I20260807 22:57:02.947685 3930 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260807 22:57:02.955116 3930 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 5afdd40906094502ab5d825fbe1315a7: tablet copy: Starting download of 0 data blocks...
I20260807 22:57:02.955530 3930 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 5afdd40906094502ab5d825fbe1315a7: tablet copy: Starting download of 1 WAL segments...
I20260807 22:57:02.959757 3930 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 5afdd40906094502ab5d825fbe1315a7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260807 22:57:02.964684 3235 tablet_copy_service.cc:342] P 9828cb5c031840cfac64207adc69ca5b: Request end of tablet copy session 5afdd40906094502ab5d825fbe1315a7-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.25.94.61@KRBTEST.COM'} at 127.0.0.1:41326
I20260807 22:57:02.964994 3235 tablet_copy_service.cc:433] P 9828cb5c031840cfac64207adc69ca5b: ending tablet copy session 5afdd40906094502ab5d825fbe1315a7-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 5afdd40906094502ab5d825fbe1315a7
I20260807 22:57:02.977965 3930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:57:02.980832 4119 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:57:02.987365 4120 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:57:02.990994 4122 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:57:02.992539 3930 server_base.cc:1034] running on GCE node
I20260807 22:57:02.992933 3930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:57:02.993721 3930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:57:02.994954 3930 hybrid_clock.cc:648] HybridClock initialized: now 1786143422994885 us; error 40 us; skew 500 ppm
I20260807 22:57:02.995190 3930 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 22:57:02 dist-test-slave-z8x9 krb5kdc[3141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143422, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.94.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:57:02.998072 3930 init.cc:377] Logged in from keytab as oryx/127.25.94.61@KRBTEST.COM (short username oryx)
I20260807 22:57:03.002264 3930 webserver.cc:460] Webserver started at http://127.25.94.61:42397/ using document root <none> and password file <none>
I20260807 22:57:03.002694 3930 fs_manager.cc:359] Metadata directory not provided
I20260807 22:57:03.002794 3930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:57:03.006240 3930 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.004s
I20260807 22:57:03.008437 4134 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:57:03.009155 3930 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260807 22:57:03.009434 3930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/master-1/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/master-1/wal
uuid: "5afdd40906094502ab5d825fbe1315a7"
format_stamp: "Formatted at 2026-08-07 22:57:02 on dist-test-slave-z8x9"
server_key: "36e3a87d77982aa803a3d0806a32d898"
server_key_iv: "8d3d5c7c6213dfb713b52a7a495e706d"
server_key_version: "encryptionkey@0"
I20260807 22:57:03.009616 3930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:57:03.034461 3930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:57:03.038900 3930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:57:03.069402 3930 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.61:45205
I20260807 22:57:03.069432 4222 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.61:45205 every 8 connection(s)
I20260807 22:57:03.070384 3930 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/master-1/data/info.pb
I20260807 22:57:03.077602 4226 sys_catalog.cc:263] Verifying existing consensus state
I20260807 22:57:03.082479 4226 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5afdd40906094502ab5d825fbe1315a7: Bootstrap starting.
I20260807 22:57:03.103396 4226 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5afdd40906094502ab5d825fbe1315a7: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 22:57:03.103950 4226 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5afdd40906094502ab5d825fbe1315a7: Bootstrap complete.
I20260807 22:57:03.104852 4226 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5afdd40906094502ab5d825fbe1315a7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9828cb5c031840cfac64207adc69ca5b" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 32777 } } peers { permanent_uuid: "5afdd40906094502ab5d825fbe1315a7" member_type: NON_VOTER last_known_addr { host: "127.25.94.61" port: 45205 } attrs { promote: true } }
I20260807 22:57:03.105103 4226 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5afdd40906094502ab5d825fbe1315a7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5afdd40906094502ab5d825fbe1315a7, State: Initialized, Role: LEARNER
I20260807 22:57:03.105383 4226 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5afdd40906094502ab5d825fbe1315a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9828cb5c031840cfac64207adc69ca5b" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 32777 } } peers { permanent_uuid: "5afdd40906094502ab5d825fbe1315a7" member_type: NON_VOTER last_known_addr { host: "127.25.94.61" port: 45205 } attrs { promote: true } }
I20260807 22:57:03.106235 4229 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5afdd40906094502ab5d825fbe1315a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9828cb5c031840cfac64207adc69ca5b" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 32777 } } peers { permanent_uuid: "5afdd40906094502ab5d825fbe1315a7" member_type: NON_VOTER last_known_addr { host: "127.25.94.61" port: 45205 } attrs { promote: true } } }
I20260807 22:57:03.106570 4226 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5afdd40906094502ab5d825fbe1315a7 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 22:57:03.106628 4229 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5afdd40906094502ab5d825fbe1315a7 [sys.catalog]: This master's current role is: LEARNER
I20260807 22:57:03.118677 4246 catalog_manager.cc:1259] Loaded cluster ID: b39391c33b034542a865d23e1ca84d55
I20260807 22:57:03.118803 4246 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5afdd40906094502ab5d825fbe1315a7: loading cluster ID for follower catalog manager: success
I20260807 22:57:03.122565 4246 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5afdd40906094502ab5d825fbe1315a7: acquiring CA information for follower catalog manager: success
I20260807 22:57:03.125761 4246 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5afdd40906094502ab5d825fbe1315a7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260807 22:57:03.168700 3920 heartbeater.cc:503] Master 127.25.94.62:32777 was elected leader, sending a full tablet report...
W20260807 22:57:03.296309 4102 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:57:03.296710 4102 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:57:03.296880 4102 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:57:03.306735 4102 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 22:57:03.306985 4102 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:57:03.307142 4102 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:57:03.307382 4102 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 22:57:03.307534 4102 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 22:57:03.320653 4102 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:33185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.25.94.62:32777,127.25.94.61:45205,127.25.94.60:40149
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/krb5kdc/oryx_127.25.94.60.keytab
--principal=oryx/127.25.94.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.60:40149
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.25.94.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 22:57:03.322203 4102 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:57:03.323616 4102 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:57:03.338526 4254 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:57:03.338495 4255 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:57:03.338544 4257 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:57:03.339938 4102 server_base.cc:1034] running on GCE node
I20260807 22:57:03.341013 4102 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:57:03.342657 4102 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:57:03.343966 4102 hybrid_clock.cc:648] HybridClock initialized: now 1786143423343915 us; error 40 us; skew 500 ppm
I20260807 22:57:03.344494 4102 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 22:57:03 dist-test-slave-z8x9 krb5kdc[3141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143423, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.94.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:57:03.350169 4102 init.cc:377] Logged in from keytab as oryx/127.25.94.60@KRBTEST.COM (short username oryx)
I20260807 22:57:03.352268 4102 webserver.cc:460] Webserver started at http://127.25.94.60:41193/ using document root <none> and password file <none>
I20260807 22:57:03.353097 4102 fs_manager.cc:359] Metadata directory not provided
I20260807 22:57:03.353320 4102 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:57:03.353693 4102 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:57:03.358044 4102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/master-2/data/instance:
uuid: "6b8de2d663914806a8d5fc4fb0eda2b9"
format_stamp: "Formatted at 2026-08-07 22:57:03 on dist-test-slave-z8x9"
server_key: "ba68823879e931aecc9370bacaba15d3"
server_key_iv: "e9e429d426b59a584cd5e116086cde6d"
server_key_version: "encryptionkey@0"
I20260807 22:57:03.358996 4102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/master-2/wal/instance:
uuid: "6b8de2d663914806a8d5fc4fb0eda2b9"
format_stamp: "Formatted at 2026-08-07 22:57:03 on dist-test-slave-z8x9"
server_key: "ba68823879e931aecc9370bacaba15d3"
server_key_iv: "e9e429d426b59a584cd5e116086cde6d"
server_key_version: "encryptionkey@0"
I20260807 22:57:03.365617 4102 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260807 22:57:03.370558 4267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:57:03.372342 4102 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260807 22:57:03.372617 4102 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/master-2/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/master-2/wal
uuid: "6b8de2d663914806a8d5fc4fb0eda2b9"
format_stamp: "Formatted at 2026-08-07 22:57:03 on dist-test-slave-z8x9"
server_key: "ba68823879e931aecc9370bacaba15d3"
server_key_iv: "e9e429d426b59a584cd5e116086cde6d"
server_key_version: "encryptionkey@0"
I20260807 22:57:03.372974 4102 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:57:03.389437 4102 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:57:03.395768 4102 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:57:03.396085 4102 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:57:03.430161 4102 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.60:40149
I20260807 22:57:03.430335 4349 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.60:40149 every 8 connection(s)
I20260807 22:57:03.432390 4102 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/master-2/data/info.pb
I20260807 22:57:03.433701 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 4102
I20260807 22:57:03.446312 4354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:57:03.474255 4357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:03.435490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.61:45205 (local address 127.0.0.1:57832)
0807 22:57:03.461464 (+ 25974us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:03.461485 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 22:57:03.461670 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:57:03.463271 (+ 1601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:57:03.463276 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:57:03.463305 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:57:03.463681 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:03.463694 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:03.467006 (+ 3312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:03.467022 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:03.469742 (+ 2720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:03.469767 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:03.472659 (+ 2892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:03.472666 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:03.472982 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:03.473050 (+ 68us) client_negotiation.cc:769] Sending connection context
0807 22:57:03.473226 (+ 176us) client_negotiation.cc:240] Negotiation successful
0807 22:57:03.473511 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25811,"thread_start_us":31224,"threads_started":1}
I20260807 22:57:03.475647 4359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:03.436271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57832 (local address 127.25.94.61:45205)
0807 22:57:03.459374 (+ 23103us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:03.459386 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:03.461800 (+ 2414us) server_negotiation.cc:407] Connection header received
0807 22:57:03.461924 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:03.461931 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:03.462062 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:03.462190 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:57:03.463858 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:03.466591 (+ 2733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:03.470071 (+ 3480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:03.472019 (+ 1948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:03.472271 (+ 252us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:03.472506 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0807 22:57:03.474621 (+ 2115us) server_negotiation.cc:299] Negotiation successful
0807 22:57:03.474890 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22796,"thread_start_us":23091,"threads_started":1}
I20260807 22:57:03.477779 4175 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 5afdd40906094502ab5d825fbe1315a7 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260807 22:57:03.478941 4229 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5afdd40906094502ab5d825fbe1315a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9828cb5c031840cfac64207adc69ca5b. Latest consensus state: current_term: 1 leader_uuid: "9828cb5c031840cfac64207adc69ca5b" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9828cb5c031840cfac64207adc69ca5b" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 32777 } } peers { permanent_uuid: "5afdd40906094502ab5d825fbe1315a7" member_type: NON_VOTER last_known_addr { host: "127.25.94.61" port: 45205 } attrs { promote: true } } }
I20260807 22:57:03.479326 4229 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5afdd40906094502ab5d825fbe1315a7 [sys.catalog]: This master's current role is: LEARNER
I20260807 22:57:03.482048 4354 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.94.62" port: 32777 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 22:57:03 dist-test-slave-z8x9 krb5kdc[3141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143419, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.25.94.60@KRBTEST.COM
Aug 07 22:57:03 dist-test-slave-z8x9 krb5kdc[3141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143423, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.94.60@KRBTEST.COM for oryx/127.25.94.62@KRBTEST.COM
I20260807 22:57:03.514532 4362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:03.487176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41336 (local address 127.25.94.62:32777)
0807 22:57:03.488226 (+ 1050us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:03.488234 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:03.489989 (+ 1755us) server_negotiation.cc:407] Connection header received
0807 22:57:03.490256 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:03.490262 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:03.490339 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:03.490454 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:03.492565 (+ 2111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:03.493933 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:03.496994 (+ 3061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:03.497839 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:03.497954 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:03.503946 (+ 5992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:57:03.503957 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:57:03.503962 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:57:03.504028 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:57:03.509978 (+ 5950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:03.511324 (+ 1346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:03.511330 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:03.511333 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:03.511563 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:03.512024 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:03.512030 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:03.512033 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:03.512550 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:57:03.512708 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0807 22:57:03.513611 (+ 903us) server_negotiation.cc:299] Negotiation successful
0807 22:57:03.513856 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":822,"thread_start_us":693,"threads_started":1}
I20260807 22:57:03.514961 4360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:03.437119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52362 (local address 127.25.94.60:40149)
0807 22:57:03.480190 (+ 43071us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:03.480207 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:03.480278 (+ 71us) server_negotiation.cc:407] Connection header received
0807 22:57:03.483284 (+ 3006us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:03.483312 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:03.483986 (+ 674us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:03.484459 (+ 473us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:03.487853 (+ 3394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:03.491063 (+ 3210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:03.492984 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:03.493960 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:03.494280 (+ 320us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:03.502938 (+ 8658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:57:03.502951 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:57:03.502966 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:57:03.503000 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:57:03.508899 (+ 5899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:03.510176 (+ 1277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:03.510183 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:03.510186 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:03.510489 (+ 303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:03.511187 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:03.511192 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:03.511194 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:03.512082 (+ 888us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:57:03.512632 (+ 550us) server_negotiation.cc:1035] Waiting for connection context
0807 22:57:03.512975 (+ 343us) server_negotiation.cc:299] Negotiation successful
0807 22:57:03.513476 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33352,"thread_start_us":43901,"threads_started":1}
I20260807 22:57:03.515082 4363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:03.487070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:32777 (local address 127.0.0.1:41336)
0807 22:57:03.489713 (+ 2643us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:03.489754 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0807 22:57:03.490104 (+ 350us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:57:03.490580 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:57:03.490602 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:57:03.491709 (+ 1107us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:57:03.492363 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:03.492373 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:03.494069 (+ 1696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:03.494077 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:03.496830 (+ 2753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:03.496848 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:03.498284 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:03.498295 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:03.498612 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:03.499733 (+ 1121us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:57:03.499818 (+ 85us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:57:03.503607 (+ 3789us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:57:03.510145 (+ 6538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:57:03.510154 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:57:03.510164 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:57:03.511185 (+ 1021us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:57:03.511686 (+ 501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:57:03.511689 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:57:03.511691 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:57:03.511908 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:57:03.512721 (+ 813us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:57:03.512730 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:57:03.513074 (+ 344us) client_negotiation.cc:769] Sending connection context
0807 22:57:03.513384 (+ 310us) client_negotiation.cc:240] Negotiation successful
0807 22:57:03.513731 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2289,"thread_start_us":1885,"threads_started":1}
I20260807 22:57:03.517895 4354 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.94.61" port: 45205 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 22:57:03 dist-test-slave-z8x9 krb5kdc[3141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143423, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.94.60@KRBTEST.COM for oryx/127.25.94.61@KRBTEST.COM
I20260807 22:57:03.537853 4363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:03.519216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.61:45205 (local address 127.0.0.1:57834)
0807 22:57:03.519477 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:03.519498 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 22:57:03.519612 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:57:03.519956 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:57:03.519960 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:57:03.520522 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:57:03.520789 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:03.520798 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:03.522304 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:03.522312 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:03.523775 (+ 1463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:03.523789 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:03.524883 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:03.524891 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:03.525232 (+ 341us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:03.526065 (+ 833us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:57:03.526083 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:57:03.529096 (+ 3013us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:57:03.534321 (+ 5225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:57:03.534326 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:57:03.534331 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:57:03.534952 (+ 621us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:57:03.535436 (+ 484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:57:03.535441 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:57:03.535444 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:57:03.535644 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:57:03.536322 (+ 678us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:57:03.536327 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:57:03.536513 (+ 186us) client_negotiation.cc:769] Sending connection context
0807 22:57:03.536663 (+ 150us) client_negotiation.cc:240] Negotiation successful
0807 22:57:03.537091 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260807 22:57:03.537853 4359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:03.519359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57834 (local address 127.25.94.61:45205)
0807 22:57:03.519575 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:03.519581 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:03.519600 (+ 19us) server_negotiation.cc:407] Connection header received
0807 22:57:03.519761 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:03.519765 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:03.519831 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:03.519935 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:03.520938 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:03.522035 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:03.523960 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:03.524654 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:03.524811 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:03.529442 (+ 4631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:57:03.529448 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:57:03.529452 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:57:03.529487 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:57:03.534180 (+ 4693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:03.535081 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:03.535087 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:03.535091 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:03.535303 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:03.535764 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:03.535771 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:03.535774 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:03.536185 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:57:03.536347 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0807 22:57:03.536767 (+ 420us) server_negotiation.cc:299] Negotiation successful
0807 22:57:03.537091 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260807 22:57:03.539568 4354 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.94.60" port: 40149 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 22:57:03 dist-test-slave-z8x9 krb5kdc[3141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143423, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.94.60@KRBTEST.COM for oryx/127.25.94.60@KRBTEST.COM
I20260807 22:57:03.558914 4360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:03.540991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52370 (local address 127.25.94.60:40149)
0807 22:57:03.541235 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:03.541241 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:03.541511 (+ 270us) server_negotiation.cc:407] Connection header received
0807 22:57:03.541700 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:03.541705 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:03.541766 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:03.541862 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:03.543067 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:03.544094 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:03.545689 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:03.546339 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:03.546501 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:03.551272 (+ 4771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:57:03.551279 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:57:03.551282 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:57:03.551314 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:57:03.555208 (+ 3894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:03.556065 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:03.556074 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:03.556076 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:03.556297 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:03.556829 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:03.556838 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:03.556840 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:03.557340 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:57:03.557495 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0807 22:57:03.557966 (+ 471us) server_negotiation.cc:299] Negotiation successful
0807 22:57:03.558359 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":114,"server-negotiator.queue_time_us":70}
I20260807 22:57:03.558935 4363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:03.541134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.60:40149 (local address 127.0.0.1:52370)
0807 22:57:03.541421 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:03.541437 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 22:57:03.541578 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:57:03.541880 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:57:03.541929 (+ 49us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:57:03.542623 (+ 694us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:57:03.542927 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:03.542937 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:03.544221 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:03.544229 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:03.545551 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:03.545563 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:03.546522 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:03.546528 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:03.546867 (+ 339us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:03.547707 (+ 840us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:57:03.547743 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:57:03.550987 (+ 3244us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:57:03.555396 (+ 4409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:57:03.555404 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:57:03.555408 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:57:03.555929 (+ 521us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:57:03.556424 (+ 495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:57:03.556429 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:57:03.556432 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:57:03.556699 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:57:03.557452 (+ 753us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:57:03.557457 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:57:03.557632 (+ 175us) client_negotiation.cc:769] Sending connection context
0807 22:57:03.557834 (+ 202us) client_negotiation.cc:240] Negotiation successful
0807 22:57:03.558149 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260807 22:57:03.566310 4354 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b8de2d663914806a8d5fc4fb0eda2b9: Bootstrap starting.
I20260807 22:57:03.571621 4354 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6b8de2d663914806a8d5fc4fb0eda2b9: Neither blocks nor log segments found. Creating new log.
I20260807 22:57:03.574041 4354 log.cc:824] T 00000000000000000000000000000000 P 6b8de2d663914806a8d5fc4fb0eda2b9: Log is configured to *not* fsync() on all Append() calls
I20260807 22:57:03.577942 4354 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b8de2d663914806a8d5fc4fb0eda2b9: No bootstrap required, opened a new log
I20260807 22:57:03.589877 4354 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6b8de2d663914806a8d5fc4fb0eda2b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9828cb5c031840cfac64207adc69ca5b" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 32777 } } peers { permanent_uuid: "5afdd40906094502ab5d825fbe1315a7" member_type: VOTER last_known_addr { host: "127.25.94.61" port: 45205 } } peers { permanent_uuid: "6b8de2d663914806a8d5fc4fb0eda2b9" member_type: VOTER last_known_addr { host: "127.25.94.60" port: 40149 } }
I20260807 22:57:03.590500 4354 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6b8de2d663914806a8d5fc4fb0eda2b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:57:03.590744 4354 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6b8de2d663914806a8d5fc4fb0eda2b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b8de2d663914806a8d5fc4fb0eda2b9, State: Initialized, Role: FOLLOWER
I20260807 22:57:03.591419 4354 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6b8de2d663914806a8d5fc4fb0eda2b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9828cb5c031840cfac64207adc69ca5b" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 32777 } } peers { permanent_uuid: "5afdd40906094502ab5d825fbe1315a7" member_type: VOTER last_known_addr { host: "127.25.94.61" port: 45205 } } peers { permanent_uuid: "6b8de2d663914806a8d5fc4fb0eda2b9" member_type: VOTER last_known_addr { host: "127.25.94.60" port: 40149 } }
I20260807 22:57:03.593063 4365 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b8de2d663914806a8d5fc4fb0eda2b9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9828cb5c031840cfac64207adc69ca5b" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 32777 } } peers { permanent_uuid: "5afdd40906094502ab5d825fbe1315a7" member_type: VOTER last_known_addr { host: "127.25.94.61" port: 45205 } } peers { permanent_uuid: "6b8de2d663914806a8d5fc4fb0eda2b9" member_type: VOTER last_known_addr { host: "127.25.94.60" port: 40149 } } }
I20260807 22:57:03.593894 4365 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b8de2d663914806a8d5fc4fb0eda2b9 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 22:57:03.594288 4354 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6b8de2d663914806a8d5fc4fb0eda2b9 [sys.catalog]: configured and running, proceeding with master startup.
W20260807 22:57:03.613569 4391 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6b8de2d663914806a8d5fc4fb0eda2b9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 22:57:03.613886 4391 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 22:57:03.618789 4102 master_runner.cc:428] Detected that this master 6b8de2d663914806a8d5fc4fb0eda2b9 is joining an existing cluster
I20260807 22:57:03.619014 4102 master_runner.cc:432] Initiating AddMaster RPC to add 127.25.94.60:40149
I20260807 22:57:03.622766 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 3303
Aug 07 22:57:03 dist-test-slave-z8x9 krb5kdc[3141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143419, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.25.94.1@KRBTEST.COM
W20260807 22:57:03.648639 4276 tablet.cc:2369] T 00000000000000000000000000000000 P 6b8de2d663914806a8d5fc4fb0eda2b9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 22:57:03.661600 4397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:03.624749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45526 (local address 127.25.94.1:38459)
0807 22:57:03.626160 (+ 1411us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:03.626167 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:03.626193 (+ 26us) server_negotiation.cc:407] Connection header received
0807 22:57:03.626266 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:03.626272 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:03.626513 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:03.626691 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:03.628993 (+ 2302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:03.631434 (+ 2441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:03.633982 (+ 2548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:03.635319 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:03.635480 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:03.648825 (+ 13345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:57:03.648854 (+ 29us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:57:03.648877 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:57:03.648930 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:57:03.655569 (+ 6639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:03.657433 (+ 1864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:03.657448 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:03.657454 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:03.657796 (+ 342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:03.658427 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:03.658433 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:03.658436 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:03.659029 (+ 593us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:57:03.659517 (+ 488us) server_negotiation.cc:1035] Waiting for connection context
0807 22:57:03.660451 (+ 934us) server_negotiation.cc:299] Negotiation successful
0807 22:57:03.660798 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1210,"thread_start_us":922,"threads_started":1}
I20260807 22:57:03.664772 3496 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 22:57:03.705796 4362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:03.638147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41342 (local address 127.25.94.62:32777)
0807 22:57:03.638438 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:03.638446 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:03.668765 (+ 30319us) server_negotiation.cc:407] Connection header received
0807 22:57:03.669418 (+ 653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:03.669424 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:03.669673 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:03.669794 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:03.673554 (+ 3760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:03.675154 (+ 1600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:03.680000 (+ 4846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:03.681131 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:03.681283 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:03.689990 (+ 8707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:57:03.690000 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:57:03.690005 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:57:03.690052 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:57:03.696014 (+ 5962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:03.697281 (+ 1267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:03.697288 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:03.697291 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:03.697559 (+ 268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:03.698308 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:03.698315 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:03.698319 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:03.698841 (+ 522us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:57:03.699131 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0807 22:57:03.700770 (+ 1639us) server_negotiation.cc:299] Negotiation successful
0807 22:57:03.705099 (+ 4329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120}
I20260807 22:57:03.711133 4403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:03.639449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:32777 (local address 127.0.0.1:41342)
0807 22:57:03.668559 (+ 29110us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:03.668576 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 22:57:03.669139 (+ 563us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:57:03.670269 (+ 1130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:57:03.670279 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:57:03.672162 (+ 1883us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:57:03.673354 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:03.673362 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:03.675417 (+ 2055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:03.675423 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:03.679826 (+ 4403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:03.679842 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:03.681890 (+ 2048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:03.681918 (+ 28us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:03.682320 (+ 402us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:03.684075 (+ 1755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:57:03.684100 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:57:03.687994 (+ 3894us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:57:03.696297 (+ 8303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:57:03.696302 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:57:03.696306 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:57:03.697041 (+ 735us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:57:03.697733 (+ 692us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:57:03.697738 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:57:03.697741 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:57:03.698043 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:57:03.699064 (+ 1021us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:57:03.699069 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:57:03.700350 (+ 1281us) client_negotiation.cc:769] Sending connection context
0807 22:57:03.708114 (+ 7764us) client_negotiation.cc:240] Negotiation successful
0807 22:57:03.709405 (+ 1291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28518,"mutex_wait_us":22,"thread_start_us":31197,"threads_started":1}
I20260807 22:57:03.714206 4360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:03.646185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52382 (local address 127.25.94.60:40149)
0807 22:57:03.646506 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:03.646521 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:03.687329 (+ 40808us) server_negotiation.cc:407] Connection header received
0807 22:57:03.687655 (+ 326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:03.687662 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:03.687731 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:03.687799 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:03.691223 (+ 3424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:03.692861 (+ 1638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:03.695999 (+ 3138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:03.696992 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:03.697090 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:03.702316 (+ 5226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:57:03.702323 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:57:03.702328 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:57:03.702361 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:57:03.707867 (+ 5506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:03.709969 (+ 2102us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:03.709977 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:03.709981 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:03.710258 (+ 277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:03.711168 (+ 910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:03.711173 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:03.711176 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:03.712185 (+ 1009us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:57:03.712459 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0807 22:57:03.713289 (+ 830us) server_negotiation.cc:299] Negotiation successful
0807 22:57:03.713594 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260807 22:57:03.717192 4359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:03.638451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57848 (local address 127.25.94.61:45205)
0807 22:57:03.638693 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:03.638700 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:03.668506 (+ 29806us) server_negotiation.cc:407] Connection header received
0807 22:57:03.668894 (+ 388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:03.668902 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:03.668999 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:03.669099 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:03.673556 (+ 4457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:03.675159 (+ 1603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:03.679923 (+ 4764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:03.681108 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:03.681271 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:03.698394 (+ 17123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:57:03.698402 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:57:03.698406 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:57:03.698445 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:57:03.704819 (+ 6374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:03.707038 (+ 2219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:03.707045 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:03.707052 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:03.707384 (+ 332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:03.708674 (+ 1290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:03.708680 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:03.708684 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:03.709223 (+ 539us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:57:03.709381 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0807 22:57:03.715943 (+ 6562us) server_negotiation.cc:299] Negotiation successful
0807 22:57:03.716251 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260807 22:57:03.717919 4408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:03.656756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.60:40149 (local address 127.0.0.1:52382)
0807 22:57:03.686588 (+ 29832us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:03.687221 (+ 633us) client_negotiation.cc:174] Beginning negotiation
0807 22:57:03.687508 (+ 287us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:57:03.688107 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:57:03.688112 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:57:03.689785 (+ 1673us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:57:03.691072 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:03.691089 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:03.693188 (+ 2099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:03.693194 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:03.695857 (+ 2663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:03.695874 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:03.697640 (+ 1766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:03.697648 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:03.697996 (+ 348us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:03.700347 (+ 2351us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:57:03.700370 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:57:03.702087 (+ 1717us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:57:03.708008 (+ 5921us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:57:03.708013 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:57:03.708017 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:57:03.709819 (+ 1802us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:57:03.710460 (+ 641us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:57:03.710465 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:57:03.710469 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:57:03.711041 (+ 572us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:57:03.712564 (+ 1523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:57:03.712569 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:57:03.712862 (+ 293us) client_negotiation.cc:769] Sending connection context
0807 22:57:03.714780 (+ 1918us) client_negotiation.cc:240] Negotiation successful
0807 22:57:03.715221 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29414,"thread_start_us":25832,"threads_started":1}
I20260807 22:57:03.718595 4401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:03.639580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.61:45205 (local address 127.0.0.1:57848)
0807 22:57:03.668338 (+ 28758us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:03.668363 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0807 22:57:03.668698 (+ 335us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:57:03.669404 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:57:03.669409 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:57:03.672114 (+ 2705us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:57:03.673325 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:03.673348 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:03.675432 (+ 2084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:03.675436 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:03.679664 (+ 4228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:03.679695 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:03.682049 (+ 2354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:03.682068 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:03.682738 (+ 670us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:03.686171 (+ 3433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:57:03.686227 (+ 56us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:57:03.689590 (+ 3363us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:57:03.706183 (+ 16593us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:57:03.706192 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:57:03.706197 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:57:03.706872 (+ 675us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:57:03.707593 (+ 721us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:57:03.707598 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:57:03.707601 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:57:03.708006 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:57:03.709350 (+ 1344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:57:03.709355 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:57:03.710210 (+ 855us) client_negotiation.cc:769] Sending connection context
0807 22:57:03.715739 (+ 5529us) client_negotiation.cc:240] Negotiation successful
0807 22:57:03.716408 (+ 669us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28506,"mutex_wait_us":1,"spinlock_wait_cycles":30080,"thread_start_us":24213,"threads_started":1}
Aug 07 22:57:03 dist-test-slave-z8x9 krb5kdc[3141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143420, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.94.62@KRBTEST.COM for oryx/127.25.94.60@KRBTEST.COM
I20260807 22:57:03.744609 4360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:03.721885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52384 (local address 127.25.94.60:40149)
0807 22:57:03.722504 (+ 619us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:03.722510 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:03.722525 (+ 15us) server_negotiation.cc:407] Connection header received
0807 22:57:03.722610 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:03.722615 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:03.722761 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:03.722855 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:03.724252 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:03.725933 (+ 1681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:03.728316 (+ 2383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:03.729488 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:03.729608 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:03.735644 (+ 6036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:57:03.735652 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:57:03.735657 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:57:03.735709 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:57:03.740569 (+ 4860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:03.741821 (+ 1252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:03.741827 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:03.741831 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:03.741981 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:03.742598 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:03.742604 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:03.742607 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:03.743054 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:57:03.743189 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0807 22:57:03.743809 (+ 620us) server_negotiation.cc:299] Negotiation successful
0807 22:57:03.744041 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":505}
I20260807 22:57:03.744895 4357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:03.721551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.60:40149 (local address 127.0.0.1:52384)
0807 22:57:03.722171 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:03.722191 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 22:57:03.722298 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:57:03.723060 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:57:03.723065 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:57:03.723732 (+ 667us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:57:03.724042 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:03.724055 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:03.726092 (+ 2037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:03.726098 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:03.728145 (+ 2047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:03.728161 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:03.729943 (+ 1782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:03.729949 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:03.730244 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:03.731219 (+ 975us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:57:03.731242 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:57:03.735358 (+ 4116us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:57:03.740746 (+ 5388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:57:03.740754 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:57:03.740758 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:57:03.741672 (+ 914us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:57:03.742143 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:57:03.742148 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:57:03.742152 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:57:03.742439 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:57:03.743247 (+ 808us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:57:03.743253 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:57:03.743481 (+ 228us) client_negotiation.cc:769] Sending connection context
0807 22:57:03.743672 (+ 191us) client_negotiation.cc:240] Negotiation successful
0807 22:57:03.744068 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":393}
I20260807 22:57:03.747255 3206 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.25.94.60:40149: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "6b8de2d663914806a8d5fc4fb0eda2b9"
member_type: NON_VOTER
last_known_addr {
host: "127.25.94.60"
port: 40149
}
attrs {
promote: true
}
}
dest_uuid: "9828cb5c031840cfac64207adc69ca5b"
cas_config_opid_index: 5
I20260807 22:57:03.748648 3206 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9828cb5c031840cfac64207adc69ca5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9828cb5c031840cfac64207adc69ca5b" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 32777 } } peers { permanent_uuid: "5afdd40906094502ab5d825fbe1315a7" member_type: NON_VOTER last_known_addr { host: "127.25.94.61" port: 45205 } attrs { promote: true } } peers { permanent_uuid: "6b8de2d663914806a8d5fc4fb0eda2b9" member_type: NON_VOTER last_known_addr { host: "127.25.94.60" port: 40149 } attrs { promote: true } }
I20260807 22:57:03.750708 4350 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9828cb5c031840cfac64207adc69ca5b [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 6b8de2d663914806a8d5fc4fb0eda2b9 (127.25.94.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9828cb5c031840cfac64207adc69ca5b" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 32777 } } peers { permanent_uuid: "5afdd40906094502ab5d825fbe1315a7" member_type: NON_VOTER last_known_addr { host: "127.25.94.61" port: 45205 } attrs { promote: true } } peers { permanent_uuid: "6b8de2d663914806a8d5fc4fb0eda2b9" member_type: NON_VOTER last_known_addr { host: "127.25.94.60" port: 40149 } attrs { promote: true } } }
I20260807 22:57:03.752076 4315 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6b8de2d663914806a8d5fc4fb0eda2b9 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:57:03.754791 4175 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5afdd40906094502ab5d825fbe1315a7 [term 1 LEARNER]: Refusing update from remote peer 9828cb5c031840cfac64207adc69ca5b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260807 22:57:03.758103 4315 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6b8de2d663914806a8d5fc4fb0eda2b9 [term 1 FOLLOWER]: Refusing update from remote peer 9828cb5c031840cfac64207adc69ca5b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260807 22:57:03.761593 4410 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9828cb5c031840cfac64207adc69ca5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5afdd40906094502ab5d825fbe1315a7" member_type: NON_VOTER last_known_addr { host: "127.25.94.61" port: 45205 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.002s
I20260807 22:57:03.768503 4365 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6b8de2d663914806a8d5fc4fb0eda2b9 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 22:57:03.768785 4350 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.25.94.60:40149
I20260807 22:57:03.769057 4365 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6b8de2d663914806a8d5fc4fb0eda2b9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9828cb5c031840cfac64207adc69ca5b" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 32777 } } peers { permanent_uuid: "5afdd40906094502ab5d825fbe1315a7" member_type: VOTER last_known_addr { host: "127.25.94.61" port: 45205 } } peers { permanent_uuid: "6b8de2d663914806a8d5fc4fb0eda2b9" member_type: VOTER last_known_addr { host: "127.25.94.60" port: 40149 } }
I20260807 22:57:03.770140 4102 master.cc:537] Master@127.25.94.60:40149 shutting down...
I20260807 22:57:03.770095 4413 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9828cb5c031840cfac64207adc69ca5b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "9828cb5c031840cfac64207adc69ca5b" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9828cb5c031840cfac64207adc69ca5b" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 32777 } } peers { permanent_uuid: "5afdd40906094502ab5d825fbe1315a7" member_type: NON_VOTER last_known_addr { host: "127.25.94.61" port: 45205 } attrs { promote: true } } peers { permanent_uuid: "6b8de2d663914806a8d5fc4fb0eda2b9" member_type: NON_VOTER last_known_addr { host: "127.25.94.60" port: 40149 } attrs { promote: true } } }
I20260807 22:57:03.771363 4413 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9828cb5c031840cfac64207adc69ca5b [sys.catalog]: This master's current role is: LEADER
I20260807 22:57:03.768108 4412 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9828cb5c031840cfac64207adc69ca5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b8de2d663914806a8d5fc4fb0eda2b9" member_type: NON_VOTER last_known_addr { host: "127.25.94.60" port: 40149 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.004s
I20260807 22:57:03.772895 4365 leader_election.cc:288] T 00000000000000000000000000000000 P 6b8de2d663914806a8d5fc4fb0eda2b9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9828cb5c031840cfac64207adc69ca5b (127.25.94.62:32777), 5afdd40906094502ab5d825fbe1315a7 (127.25.94.61:45205)
I20260807 22:57:03.774430 4175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6b8de2d663914806a8d5fc4fb0eda2b9" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5afdd40906094502ab5d825fbe1315a7" is_pre_election: true
I20260807 22:57:03.775424 3221 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6b8de2d663914806a8d5fc4fb0eda2b9" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9828cb5c031840cfac64207adc69ca5b" is_pre_election: true
I20260807 22:57:03.777034 4268 leader_election.cc:302] T 00000000000000000000000000000000 P 6b8de2d663914806a8d5fc4fb0eda2b9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b8de2d663914806a8d5fc4fb0eda2b9; no voters: 5afdd40906094502ab5d825fbe1315a7, 9828cb5c031840cfac64207adc69ca5b
I20260807 22:57:03.778839 4365 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 6b8de2d663914806a8d5fc4fb0eda2b9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260807 22:57:03.792702 3186 consensus_peers.cc:487] T 00000000000000000000000000000000 P 9828cb5c031840cfac64207adc69ca5b -> Peer 6b8de2d663914806a8d5fc4fb0eda2b9 (127.25.94.60:40149): Couldn't send request to peer 6b8de2d663914806a8d5fc4fb0eda2b9. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260807 22:57:03.793769 4175 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5afdd40906094502ab5d825fbe1315a7 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 6b8de2d663914806a8d5fc4fb0eda2b9 (127.25.94.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9828cb5c031840cfac64207adc69ca5b" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 32777 } } peers { permanent_uuid: "5afdd40906094502ab5d825fbe1315a7" member_type: NON_VOTER last_known_addr { host: "127.25.94.61" port: 45205 } attrs { promote: true } } peers { permanent_uuid: "6b8de2d663914806a8d5fc4fb0eda2b9" member_type: NON_VOTER last_known_addr { host: "127.25.94.60" port: 40149 } attrs { promote: true } } }
I20260807 22:57:03.797968 4229 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5afdd40906094502ab5d825fbe1315a7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9828cb5c031840cfac64207adc69ca5b" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9828cb5c031840cfac64207adc69ca5b" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 32777 } } peers { permanent_uuid: "5afdd40906094502ab5d825fbe1315a7" member_type: NON_VOTER last_known_addr { host: "127.25.94.61" port: 45205 } attrs { promote: true } } peers { permanent_uuid: "6b8de2d663914806a8d5fc4fb0eda2b9" member_type: NON_VOTER last_known_addr { host: "127.25.94.60" port: 40149 } attrs { promote: true } } }
I20260807 22:57:03.798349 4229 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5afdd40906094502ab5d825fbe1315a7 [sys.catalog]: This master's current role is: LEARNER
I20260807 22:57:03.799038 4410 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 9828cb5c031840cfac64207adc69ca5b: attempting to promote NON_VOTER 5afdd40906094502ab5d825fbe1315a7 to VOTER
I20260807 22:57:03.800272 4410 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9828cb5c031840cfac64207adc69ca5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9828cb5c031840cfac64207adc69ca5b" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 32777 } } peers { permanent_uuid: "5afdd40906094502ab5d825fbe1315a7" member_type: VOTER last_known_addr { host: "127.25.94.61" port: 45205 } attrs { promote: false } } peers { permanent_uuid: "6b8de2d663914806a8d5fc4fb0eda2b9" member_type: NON_VOTER last_known_addr { host: "127.25.94.60" port: 40149 } attrs { promote: true } }
I20260807 22:57:03.803820 4175 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5afdd40906094502ab5d825fbe1315a7 [term 1 LEARNER]: Refusing update from remote peer 9828cb5c031840cfac64207adc69ca5b: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20260807 22:57:03.804231 3186 consensus_peers.cc:487] T 00000000000000000000000000000000 P 9828cb5c031840cfac64207adc69ca5b -> Peer 6b8de2d663914806a8d5fc4fb0eda2b9 (127.25.94.60:40149): Couldn't send request to peer 6b8de2d663914806a8d5fc4fb0eda2b9. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260807 22:57:03.804610 4412 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9828cb5c031840cfac64207adc69ca5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5afdd40906094502ab5d825fbe1315a7" member_type: VOTER last_known_addr { host: "127.25.94.61" port: 45205 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260807 22:57:03.807471 4413 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9828cb5c031840cfac64207adc69ca5b [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 5afdd40906094502ab5d825fbe1315a7 (127.25.94.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9828cb5c031840cfac64207adc69ca5b" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 32777 } } peers { permanent_uuid: "5afdd40906094502ab5d825fbe1315a7" member_type: VOTER last_known_addr { host: "127.25.94.61" port: 45205 } attrs { promote: false } } peers { permanent_uuid: "6b8de2d663914806a8d5fc4fb0eda2b9" member_type: NON_VOTER last_known_addr { host: "127.25.94.60" port: 40149 } attrs { promote: true } } }
I20260807 22:57:03.812194 4350 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9828cb5c031840cfac64207adc69ca5b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "9828cb5c031840cfac64207adc69ca5b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9828cb5c031840cfac64207adc69ca5b" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 32777 } } peers { permanent_uuid: "5afdd40906094502ab5d825fbe1315a7" member_type: VOTER last_known_addr { host: "127.25.94.61" port: 45205 } attrs { promote: false } } peers { permanent_uuid: "6b8de2d663914806a8d5fc4fb0eda2b9" member_type: NON_VOTER last_known_addr { host: "127.25.94.60" port: 40149 } attrs { promote: true } } }
I20260807 22:57:03.812564 4350 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9828cb5c031840cfac64207adc69ca5b [sys.catalog]: This master's current role is: LEADER
I20260807 22:57:03.812604 4102 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 6b8de2d663914806a8d5fc4fb0eda2b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 22:57:03.813212 4102 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 6b8de2d663914806a8d5fc4fb0eda2b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 22:57:03.813398 4350 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9828cb5c031840cfac64207adc69ca5b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9828cb5c031840cfac64207adc69ca5b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9828cb5c031840cfac64207adc69ca5b" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 32777 } } peers { permanent_uuid: "5afdd40906094502ab5d825fbe1315a7" member_type: VOTER last_known_addr { host: "127.25.94.61" port: 45205 } attrs { promote: false } } peers { permanent_uuid: "6b8de2d663914806a8d5fc4fb0eda2b9" member_type: NON_VOTER last_known_addr { host: "127.25.94.60" port: 40149 } attrs { promote: true } } }
I20260807 22:57:03.813580 4102 tablet_replica.cc:330] T 00000000000000000000000000000000 P 6b8de2d663914806a8d5fc4fb0eda2b9: stopping tablet replica
I20260807 22:57:03.813748 4350 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9828cb5c031840cfac64207adc69ca5b [sys.catalog]: This master's current role is: LEADER
I20260807 22:57:03.815086 4175 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5afdd40906094502ab5d825fbe1315a7 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 5afdd40906094502ab5d825fbe1315a7 (127.25.94.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9828cb5c031840cfac64207adc69ca5b" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 32777 } } peers { permanent_uuid: "5afdd40906094502ab5d825fbe1315a7" member_type: VOTER last_known_addr { host: "127.25.94.61" port: 45205 } attrs { promote: false } } peers { permanent_uuid: "6b8de2d663914806a8d5fc4fb0eda2b9" member_type: NON_VOTER last_known_addr { host: "127.25.94.60" port: 40149 } attrs { promote: true } } }
I20260807 22:57:03.821364 4229 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5afdd40906094502ab5d825fbe1315a7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9828cb5c031840cfac64207adc69ca5b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9828cb5c031840cfac64207adc69ca5b" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 32777 } } peers { permanent_uuid: "5afdd40906094502ab5d825fbe1315a7" member_type: VOTER last_known_addr { host: "127.25.94.61" port: 45205 } attrs { promote: false } } peers { permanent_uuid: "6b8de2d663914806a8d5fc4fb0eda2b9" member_type: NON_VOTER last_known_addr { host: "127.25.94.60" port: 40149 } attrs { promote: true } } }
I20260807 22:57:03.821669 4229 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5afdd40906094502ab5d825fbe1315a7 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 22:57:03.821986 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 3303
I20260807 22:57:03.849992 4102 master.cc:552] Master@127.25.94.60:40149 shutdown complete.
I20260807 22:57:03.850111 4102 master_runner.cc:305] Clearing existing system tablet
I20260807 22:57:03.850574 4102 fs_manager.cc:359] Metadata directory not provided
I20260807 22:57:03.850711 4102 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:57:03.856220 4102 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260807 22:57:03.856588 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.1:38459
--local_ip_for_outbound_sockets=127.25.94.1
--tserver_master_addrs=127.25.94.62:32777
--webserver_port=36483
--webserver_interface=127.25.94.1
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/krb5kdc/oryx_127.25.94.1.keytab
--principal=oryx/127.25.94.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.20:33185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.25.94.62:32777,127.25.94.61:45205
--tserver_master_addrs=127.25.94.62:32777,127.25.94.61:45205,127.25.94.60:40149 with env {KRB5CCNAME=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 22:57:03.858052 4420 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:57:03.858428 4102 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:57:03.858747 4102 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260807 22:57:03.858862 4102 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/master-2/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/master-2/wal
uuid: "6b8de2d663914806a8d5fc4fb0eda2b9"
format_stamp: "Formatted at 2026-08-07 22:57:03 on dist-test-slave-z8x9"
server_key: "ba68823879e931aecc9370bacaba15d3"
server_key_iv: "e9e429d426b59a584cd5e116086cde6d"
server_key_version: "encryptionkey@0"
I20260807 22:57:03.862929 4102 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 6b8de2d663914806a8d5fc4fb0eda2b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 22:57:03.868660 4102 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 6b8de2d663914806a8d5fc4fb0eda2b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260807 22:57:03.868866 4102 log.cc:1196] T 00000000000000000000000000000000 P 6b8de2d663914806a8d5fc4fb0eda2b9: Deleting WAL directory at /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260807 22:57:03.869488 4102 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 6b8de2d663914806a8d5fc4fb0eda2b9: Deleting consensus metadata
I20260807 22:57:03.869769 4102 master_runner.cc:315] Copying system tablet from 127.25.94.62:32777
I20260807 22:57:03.872370 4102 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 6b8de2d663914806a8d5fc4fb0eda2b9: tablet copy: Beginning tablet copy session from remote peer at address 127.25.94.62:32777
I20260807 22:57:03.886276 4433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:03.873403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:32777 (local address 127.0.0.1:41352)
0807 22:57:03.873982 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:03.873998 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 22:57:03.874167 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:57:03.874549 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:57:03.874554 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:57:03.874994 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:57:03.875219 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:03.875228 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:03.876274 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:03.876280 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:03.877431 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:03.877443 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:03.878164 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:03.878170 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:03.878372 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:03.879104 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:57:03.879130 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:57:03.879919 (+ 789us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:57:03.883464 (+ 3545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:57:03.883470 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:57:03.883475 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:57:03.883877 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:57:03.884309 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:57:03.884314 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:57:03.884318 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:57:03.884525 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:57:03.885111 (+ 586us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:57:03.885116 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:57:03.885370 (+ 254us) client_negotiation.cc:769] Sending connection context
0807 22:57:03.885546 (+ 176us) client_negotiation.cc:240] Negotiation successful
0807 22:57:03.885871 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":306,"threads_started":1}
I20260807 22:57:03.887192 4362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:03.873302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41352 (local address 127.25.94.62:32777)
0807 22:57:03.873574 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:03.873584 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:03.874090 (+ 506us) server_negotiation.cc:407] Connection header received
0807 22:57:03.874313 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:03.874319 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:03.874411 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:03.874519 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:03.875376 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:03.876127 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:03.877572 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:03.878004 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:03.878147 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:03.880112 (+ 1965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:57:03.880119 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:57:03.880125 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:57:03.880163 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:57:03.883313 (+ 3150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:03.884002 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:03.884008 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:03.884012 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:03.884161 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:03.884637 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:03.884645 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:03.884648 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:03.884992 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:57:03.885150 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0807 22:57:03.886706 (+ 1556us) server_negotiation.cc:299] Negotiation successful
0807 22:57:03.886930 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260807 22:57:03.887666 3235 tablet_copy_service.cc:140] P 9828cb5c031840cfac64207adc69ca5b: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 6b8de2d663914806a8d5fc4fb0eda2b9 ({username='oryx', principal='oryx/127.25.94.60@KRBTEST.COM'} at 127.0.0.1:41352)
I20260807 22:57:03.887881 3235 tablet_copy_service.cc:161] P 9828cb5c031840cfac64207adc69ca5b: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 6b8de2d663914806a8d5fc4fb0eda2b9 at {username='oryx', principal='oryx/127.25.94.60@KRBTEST.COM'} at 127.0.0.1:41352: session id = 6b8de2d663914806a8d5fc4fb0eda2b9-00000000000000000000000000000000
I20260807 22:57:03.890904 3235 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 9828cb5c031840cfac64207adc69ca5b: Tablet Copy: opened 0 blocks and 1 log segments
I20260807 22:57:03.892575 4102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260807 22:57:03.897665 4102 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 6b8de2d663914806a8d5fc4fb0eda2b9: tablet copy: Starting download of 0 data blocks...
I20260807 22:57:03.897951 4102 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 6b8de2d663914806a8d5fc4fb0eda2b9: tablet copy: Starting download of 1 WAL segments...
I20260807 22:57:03.900249 4102 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 6b8de2d663914806a8d5fc4fb0eda2b9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260807 22:57:03.903524 3235 tablet_copy_service.cc:342] P 9828cb5c031840cfac64207adc69ca5b: Request end of tablet copy session 6b8de2d663914806a8d5fc4fb0eda2b9-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.25.94.60@KRBTEST.COM'} at 127.0.0.1:41352
I20260807 22:57:03.903673 3235 tablet_copy_service.cc:433] P 9828cb5c031840cfac64207adc69ca5b: ending tablet copy session 6b8de2d663914806a8d5fc4fb0eda2b9-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 6b8de2d663914806a8d5fc4fb0eda2b9
I20260807 22:57:03.914014 4102 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:57:03.916924 4434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:57:03.927286 4102 server_base.cc:1034] running on GCE node
W20260807 22:57:03.927827 4438 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:57:03.927851 4436 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:57:03.928721 4102 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:57:03.929478 4102 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:57:03.930779 4102 hybrid_clock.cc:648] HybridClock initialized: now 1786143423930693 us; error 35 us; skew 500 ppm
I20260807 22:57:03.931152 4102 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 22:57:03 dist-test-slave-z8x9 krb5kdc[3141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143423, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.94.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:57:03.933697 4102 init.cc:377] Logged in from keytab as oryx/127.25.94.60@KRBTEST.COM (short username oryx)
I20260807 22:57:03.938279 4102 webserver.cc:460] Webserver started at http://127.25.94.60:44811/ using document root <none> and password file <none>
I20260807 22:57:03.938637 4102 fs_manager.cc:359] Metadata directory not provided
I20260807 22:57:03.938755 4102 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:57:03.941428 4102 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260807 22:57:03.943240 4447 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:57:03.943683 4102 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260807 22:57:03.943815 4102 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/master-2/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/master-2/wal
uuid: "6b8de2d663914806a8d5fc4fb0eda2b9"
format_stamp: "Formatted at 2026-08-07 22:57:03 on dist-test-slave-z8x9"
server_key: "ba68823879e931aecc9370bacaba15d3"
server_key_iv: "e9e429d426b59a584cd5e116086cde6d"
server_key_version: "encryptionkey@0"
I20260807 22:57:03.944010 4102 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:57:03.964564 4102 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:57:03.968235 4102 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:57:03.993839 4102 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.60:40149
I20260807 22:57:03.993888 4534 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.60:40149 every 8 connection(s)
I20260807 22:57:03.994747 4102 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/master-2/data/info.pb
I20260807 22:57:04.002065 4538 sys_catalog.cc:263] Verifying existing consensus state
I20260807 22:57:04.006942 4538 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b8de2d663914806a8d5fc4fb0eda2b9: Bootstrap starting.
I20260807 22:57:04.029541 4538 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b8de2d663914806a8d5fc4fb0eda2b9: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 22:57:04.030105 4538 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b8de2d663914806a8d5fc4fb0eda2b9: Bootstrap complete.
I20260807 22:57:04.031185 4538 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6b8de2d663914806a8d5fc4fb0eda2b9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9828cb5c031840cfac64207adc69ca5b" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 32777 } } peers { permanent_uuid: "5afdd40906094502ab5d825fbe1315a7" member_type: VOTER last_known_addr { host: "127.25.94.61" port: 45205 } attrs { promote: false } } peers { permanent_uuid: "6b8de2d663914806a8d5fc4fb0eda2b9" member_type: NON_VOTER last_known_addr { host: "127.25.94.60" port: 40149 } attrs { promote: true } }
I20260807 22:57:04.031534 4538 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6b8de2d663914806a8d5fc4fb0eda2b9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6b8de2d663914806a8d5fc4fb0eda2b9, State: Initialized, Role: LEARNER
I20260807 22:57:04.031819 4538 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6b8de2d663914806a8d5fc4fb0eda2b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9828cb5c031840cfac64207adc69ca5b" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 32777 } } peers { permanent_uuid: "5afdd40906094502ab5d825fbe1315a7" member_type: VOTER last_known_addr { host: "127.25.94.61" port: 45205 } attrs { promote: false } } peers { permanent_uuid: "6b8de2d663914806a8d5fc4fb0eda2b9" member_type: NON_VOTER last_known_addr { host: "127.25.94.60" port: 40149 } attrs { promote: true } }
I20260807 22:57:04.032835 4542 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b8de2d663914806a8d5fc4fb0eda2b9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9828cb5c031840cfac64207adc69ca5b" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 32777 } } peers { permanent_uuid: "5afdd40906094502ab5d825fbe1315a7" member_type: VOTER last_known_addr { host: "127.25.94.61" port: 45205 } attrs { promote: false } } peers { permanent_uuid: "6b8de2d663914806a8d5fc4fb0eda2b9" member_type: NON_VOTER last_known_addr { host: "127.25.94.60" port: 40149 } attrs { promote: true } } }
I20260807 22:57:04.033157 4538 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6b8de2d663914806a8d5fc4fb0eda2b9 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 22:57:04.033385 4542 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b8de2d663914806a8d5fc4fb0eda2b9 [sys.catalog]: This master's current role is: LEARNER
I20260807 22:57:04.043941 4559 catalog_manager.cc:1259] Loaded cluster ID: b39391c33b034542a865d23e1ca84d55
I20260807 22:57:04.044054 4559 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6b8de2d663914806a8d5fc4fb0eda2b9: loading cluster ID for follower catalog manager: success
I20260807 22:57:04.046882 4559 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6b8de2d663914806a8d5fc4fb0eda2b9: acquiring CA information for follower catalog manager: success
I20260807 22:57:04.048695 4559 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6b8de2d663914806a8d5fc4fb0eda2b9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260807 22:57:04.248061 4357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:04.234674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.60:40149 (local address 127.0.0.1:52386)
0807 22:57:04.235025 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:04.235058 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0807 22:57:04.235251 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:57:04.236496 (+ 1245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:57:04.236502 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:57:04.236584 (+ 82us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:57:04.236881 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:04.236898 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:04.239585 (+ 2687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:04.239592 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:04.242662 (+ 3070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:04.242683 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:04.246304 (+ 3621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:04.246311 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:04.246779 (+ 468us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:04.246879 (+ 100us) client_negotiation.cc:769] Sending connection context
0807 22:57:04.247098 (+ 219us) client_negotiation.cc:240] Negotiation successful
0807 22:57:04.247421 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191}
I20260807 22:57:04.248389 4562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:04.234936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52386 (local address 127.25.94.60:40149)
0807 22:57:04.236045 (+ 1109us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:04.236051 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:04.236072 (+ 21us) server_negotiation.cc:407] Connection header received
0807 22:57:04.236147 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:04.236167 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:04.236257 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:04.236382 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:57:04.237137 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:04.239329 (+ 2192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:04.242932 (+ 3603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:04.245991 (+ 3059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:04.246168 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:04.246549 (+ 381us) server_negotiation.cc:1035] Waiting for connection context
0807 22:57:04.247270 (+ 721us) server_negotiation.cc:299] Negotiation successful
0807 22:57:04.247510 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":968,"thread_start_us":744,"threads_started":1}
I20260807 22:57:04.250267 4487 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 6b8de2d663914806a8d5fc4fb0eda2b9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260807 22:57:04.251271 4542 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b8de2d663914806a8d5fc4fb0eda2b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9828cb5c031840cfac64207adc69ca5b. Latest consensus state: current_term: 1 leader_uuid: "9828cb5c031840cfac64207adc69ca5b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9828cb5c031840cfac64207adc69ca5b" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 32777 } } peers { permanent_uuid: "5afdd40906094502ab5d825fbe1315a7" member_type: VOTER last_known_addr { host: "127.25.94.61" port: 45205 } attrs { promote: false } } peers { permanent_uuid: "6b8de2d663914806a8d5fc4fb0eda2b9" member_type: NON_VOTER last_known_addr { host: "127.25.94.60" port: 40149 } attrs { promote: true } } }
I20260807 22:57:04.251899 4542 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b8de2d663914806a8d5fc4fb0eda2b9 [sys.catalog]: This master's current role is: LEARNER
W20260807 22:57:04.416586 4424 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:57:04.417012 4424 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:57:04.417348 4424 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:57:04.428236 4424 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:57:04.428500 4424 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:57:04.428790 4424 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.1
I20260807 22:57:04.445065 4424 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:33185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/krb5kdc/oryx_127.25.94.1.keytab
--principal=oryx/127.25.94.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.1:38459
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.94.1
--webserver_port=36483
--tserver_master_addrs=127.25.94.62:32777,127.25.94.61:45205,127.25.94.60:40149
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.1
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 22:57:04.446880 4424 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:57:04.448606 4424 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:57:04.469782 4573 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:57:04.470431 4568 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:57:04.470712 4569 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:57:04.472121 4424 server_base.cc:1034] running on GCE node
I20260807 22:57:04.472838 4424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:57:04.474017 4424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:57:04.475363 4424 hybrid_clock.cc:648] HybridClock initialized: now 1786143424475252 us; error 57 us; skew 500 ppm
I20260807 22:57:04.475868 4424 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 22:57:04 dist-test-slave-z8x9 krb5kdc[3141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143424, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.94.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:57:04.480914 4424 init.cc:377] Logged in from keytab as oryx/127.25.94.1@KRBTEST.COM (short username oryx)
I20260807 22:57:04.483376 4424 webserver.cc:460] Webserver started at http://127.25.94.1:36483/ using document root <none> and password file <none>
I20260807 22:57:04.484253 4424 fs_manager.cc:359] Metadata directory not provided
I20260807 22:57:04.484526 4424 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:57:04.492792 4424 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.001s
I20260807 22:57:04.497494 4584 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:57:04.499202 4424 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260807 22:57:04.499423 4424 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/ts-0/wal
uuid: "00fdb2f332f3437ea55e22ad5bf8ad14"
format_stamp: "Formatted at 2026-08-07 22:57:00 on dist-test-slave-z8x9"
server_key: "6155cdf0f39b74d11e287b90a9ef702a"
server_key_iv: "729f919dc147b53681cedae8f1d569d1"
server_key_version: "encryptionkey@0"
I20260807 22:57:04.501019 4424 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:57:04.523231 4424 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:57:04.529029 4424 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:57:04.529582 4424 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:57:04.531054 4424 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:57:04.533663 4424 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:57:04.533876 4424 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:57:04.534080 4424 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:57:04.534248 4424 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:57:04.636664 4424 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.1:38459
I20260807 22:57:04.636982 4779 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.1:38459 every 8 connection(s)
I20260807 22:57:04.639071 4424 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/ts-0/data/info.pb
I20260807 22:57:04.647172 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 4424
I20260807 22:57:04.647763 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 3515
I20260807 22:57:04.689159 4412 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9828cb5c031840cfac64207adc69ca5b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9828cb5c031840cfac64207adc69ca5b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9828cb5c031840cfac64207adc69ca5b" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 32777 } } peers { permanent_uuid: "5afdd40906094502ab5d825fbe1315a7" member_type: VOTER last_known_addr { host: "127.25.94.61" port: 45205 } attrs { promote: false } } peers { permanent_uuid: "6b8de2d663914806a8d5fc4fb0eda2b9" member_type: NON_VOTER last_known_addr { host: "127.25.94.60" port: 40149 } attrs { promote: true } } }
I20260807 22:57:04.689704 4412 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9828cb5c031840cfac64207adc69ca5b [sys.catalog]: This master's current role is: LEADER
Aug 07 22:57:04 dist-test-slave-z8x9 krb5kdc[3141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143419, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.25.94.2@KRBTEST.COM
I20260807 22:57:04.697595 4789 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 9828cb5c031840cfac64207adc69ca5b: attempting to promote NON_VOTER 6b8de2d663914806a8d5fc4fb0eda2b9 to VOTER
I20260807 22:57:04.698601 4789 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9828cb5c031840cfac64207adc69ca5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9828cb5c031840cfac64207adc69ca5b" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 32777 } } peers { permanent_uuid: "5afdd40906094502ab5d825fbe1315a7" member_type: VOTER last_known_addr { host: "127.25.94.61" port: 45205 } attrs { promote: false } } peers { permanent_uuid: "6b8de2d663914806a8d5fc4fb0eda2b9" member_type: VOTER last_known_addr { host: "127.25.94.60" port: 40149 } attrs { promote: false } }
I20260807 22:57:04.701844 4175 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5afdd40906094502ab5d825fbe1315a7 [term 1 FOLLOWER]: Refusing update from remote peer 9828cb5c031840cfac64207adc69ca5b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260807 22:57:04.702716 4487 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6b8de2d663914806a8d5fc4fb0eda2b9 [term 1 LEARNER]: Refusing update from remote peer 9828cb5c031840cfac64207adc69ca5b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260807 22:57:04.703575 4789 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9828cb5c031840cfac64207adc69ca5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5afdd40906094502ab5d825fbe1315a7" member_type: VOTER last_known_addr { host: "127.25.94.61" port: 45205 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260807 22:57:04.704372 4412 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9828cb5c031840cfac64207adc69ca5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b8de2d663914806a8d5fc4fb0eda2b9" member_type: VOTER last_known_addr { host: "127.25.94.60" port: 40149 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260807 22:57:04.707383 4789 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9828cb5c031840cfac64207adc69ca5b [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6b8de2d663914806a8d5fc4fb0eda2b9 (127.25.94.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9828cb5c031840cfac64207adc69ca5b" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 32777 } } peers { permanent_uuid: "5afdd40906094502ab5d825fbe1315a7" member_type: VOTER last_known_addr { host: "127.25.94.61" port: 45205 } attrs { promote: false } } peers { permanent_uuid: "6b8de2d663914806a8d5fc4fb0eda2b9" member_type: VOTER last_known_addr { host: "127.25.94.60" port: 40149 } attrs { promote: false } } }
I20260807 22:57:04.708416 4175 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5afdd40906094502ab5d825fbe1315a7 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6b8de2d663914806a8d5fc4fb0eda2b9 (127.25.94.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9828cb5c031840cfac64207adc69ca5b" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 32777 } } peers { permanent_uuid: "5afdd40906094502ab5d825fbe1315a7" member_type: VOTER last_known_addr { host: "127.25.94.61" port: 45205 } attrs { promote: false } } peers { permanent_uuid: "6b8de2d663914806a8d5fc4fb0eda2b9" member_type: VOTER last_known_addr { host: "127.25.94.60" port: 40149 } attrs { promote: false } } }
I20260807 22:57:04.709316 4794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:04.649415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35032 (local address 127.25.94.2:36449)
0807 22:57:04.668892 (+ 19477us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:04.668903 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:04.668944 (+ 41us) server_negotiation.cc:407] Connection header received
0807 22:57:04.669033 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:04.669042 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:04.669451 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:04.669657 (+ 206us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:04.674331 (+ 4674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:04.675907 (+ 1576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:04.678375 (+ 2468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:04.680383 (+ 2008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:04.680626 (+ 243us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:04.693126 (+ 12500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:57:04.693158 (+ 32us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:57:04.693217 (+ 59us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:57:04.693327 (+ 110us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:57:04.701668 (+ 8341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:04.703593 (+ 1925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:04.703604 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:04.703624 (+ 20us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:04.704008 (+ 384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:04.704653 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:04.704659 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:04.704662 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:04.705383 (+ 721us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:57:04.705822 (+ 439us) server_negotiation.cc:1035] Waiting for connection context
0807 22:57:04.707948 (+ 2126us) server_negotiation.cc:299] Negotiation successful
0807 22:57:04.708448 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19211,"thread_start_us":20674,"threads_started":1}
I20260807 22:57:04.711406 3707 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 22:57:04.713925 4789 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9828cb5c031840cfac64207adc69ca5b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "9828cb5c031840cfac64207adc69ca5b" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9828cb5c031840cfac64207adc69ca5b" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 32777 } } peers { permanent_uuid: "5afdd40906094502ab5d825fbe1315a7" member_type: VOTER last_known_addr { host: "127.25.94.61" port: 45205 } attrs { promote: false } } peers { permanent_uuid: "6b8de2d663914806a8d5fc4fb0eda2b9" member_type: VOTER last_known_addr { host: "127.25.94.60" port: 40149 } attrs { promote: false } } }
I20260807 22:57:04.714534 4789 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9828cb5c031840cfac64207adc69ca5b [sys.catalog]: This master's current role is: LEADER
I20260807 22:57:04.714764 4802 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9828cb5c031840cfac64207adc69ca5b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9828cb5c031840cfac64207adc69ca5b" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9828cb5c031840cfac64207adc69ca5b" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 32777 } } peers { permanent_uuid: "5afdd40906094502ab5d825fbe1315a7" member_type: VOTER last_known_addr { host: "127.25.94.61" port: 45205 } attrs { promote: false } } peers { permanent_uuid: "6b8de2d663914806a8d5fc4fb0eda2b9" member_type: VOTER last_known_addr { host: "127.25.94.60" port: 40149 } attrs { promote: false } } }
I20260807 22:57:04.715471 4802 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9828cb5c031840cfac64207adc69ca5b [sys.catalog]: This master's current role is: LEADER
I20260807 22:57:04.715667 4487 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6b8de2d663914806a8d5fc4fb0eda2b9 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6b8de2d663914806a8d5fc4fb0eda2b9 (127.25.94.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9828cb5c031840cfac64207adc69ca5b" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 32777 } } peers { permanent_uuid: "5afdd40906094502ab5d825fbe1315a7" member_type: VOTER last_known_addr { host: "127.25.94.61" port: 45205 } attrs { promote: false } } peers { permanent_uuid: "6b8de2d663914806a8d5fc4fb0eda2b9" member_type: VOTER last_known_addr { host: "127.25.94.60" port: 40149 } attrs { promote: false } } }
I20260807 22:57:04.723191 4542 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b8de2d663914806a8d5fc4fb0eda2b9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9828cb5c031840cfac64207adc69ca5b" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9828cb5c031840cfac64207adc69ca5b" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 32777 } } peers { permanent_uuid: "5afdd40906094502ab5d825fbe1315a7" member_type: VOTER last_known_addr { host: "127.25.94.61" port: 45205 } attrs { promote: false } } peers { permanent_uuid: "6b8de2d663914806a8d5fc4fb0eda2b9" member_type: VOTER last_known_addr { host: "127.25.94.60" port: 40149 } attrs { promote: false } } }
I20260807 22:57:04.723500 4542 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b8de2d663914806a8d5fc4fb0eda2b9 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 22:57:04.731012 4797 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9828cb5c031840cfac64207adc69ca5b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9828cb5c031840cfac64207adc69ca5b" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9828cb5c031840cfac64207adc69ca5b" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 32777 } } peers { permanent_uuid: "5afdd40906094502ab5d825fbe1315a7" member_type: VOTER last_known_addr { host: "127.25.94.61" port: 45205 } attrs { promote: false } } peers { permanent_uuid: "6b8de2d663914806a8d5fc4fb0eda2b9" member_type: VOTER last_known_addr { host: "127.25.94.60" port: 40149 } attrs { promote: false } } }
I20260807 22:57:04.732111 4797 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9828cb5c031840cfac64207adc69ca5b [sys.catalog]: This master's current role is: LEADER
I20260807 22:57:04.733722 4812 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5afdd40906094502ab5d825fbe1315a7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9828cb5c031840cfac64207adc69ca5b" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9828cb5c031840cfac64207adc69ca5b" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 32777 } } peers { permanent_uuid: "5afdd40906094502ab5d825fbe1315a7" member_type: VOTER last_known_addr { host: "127.25.94.61" port: 45205 } attrs { promote: false } } peers { permanent_uuid: "6b8de2d663914806a8d5fc4fb0eda2b9" member_type: VOTER last_known_addr { host: "127.25.94.60" port: 40149 } attrs { promote: false } } }
Aug 07 22:57:04 dist-test-slave-z8x9 krb5kdc[3141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143424, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.94.1@KRBTEST.COM for oryx/127.25.94.62@KRBTEST.COM
I20260807 22:57:04.735838 4812 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5afdd40906094502ab5d825fbe1315a7 [sys.catalog]: This master's current role is: FOLLOWER
Aug 07 22:57:04 dist-test-slave-z8x9 krb5kdc[3141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143424, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.94.1@KRBTEST.COM for oryx/127.25.94.60@KRBTEST.COM
Aug 07 22:57:04 dist-test-slave-z8x9 krb5kdc[3141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143424, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.94.1@KRBTEST.COM for oryx/127.25.94.61@KRBTEST.COM
I20260807 22:57:04.750160 4562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:04.648326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.1:60465 (local address 127.25.94.60:40149)
0807 22:57:04.648578 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:04.648587 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:04.689906 (+ 41319us) server_negotiation.cc:407] Connection header received
0807 22:57:04.696013 (+ 6107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:04.696023 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:04.696148 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:04.696325 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:04.703455 (+ 7130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:04.704948 (+ 1493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:04.729046 (+ 24098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:04.730455 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:04.730590 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:04.738888 (+ 8298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:57:04.738894 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:57:04.738899 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:57:04.738949 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:57:04.745041 (+ 6092us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:04.746497 (+ 1456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:04.746504 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:04.746508 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:04.746840 (+ 332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:04.747578 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:04.747584 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:04.747588 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:04.747997 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:57:04.748128 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0807 22:57:04.749237 (+ 1109us) server_negotiation.cc:299] Negotiation successful
0807 22:57:04.749567 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260807 22:57:04.751765 4787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:04.645824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.1:41937 (local address 127.25.94.62:32777)
0807 22:57:04.678318 (+ 32494us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:04.678327 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:04.691163 (+ 12836us) server_negotiation.cc:407] Connection header received
0807 22:57:04.695522 (+ 4359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:04.695533 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:04.695671 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:04.695825 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:04.713622 (+ 17797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:04.715621 (+ 1999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:04.726713 (+ 11092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:04.727939 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:04.728069 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:04.738873 (+ 10804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:57:04.738881 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:57:04.738885 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:57:04.738940 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:57:04.745670 (+ 6730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:04.747400 (+ 1730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:04.747408 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:04.747414 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:04.747666 (+ 252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:04.748568 (+ 902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:04.748575 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:04.748579 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:04.749300 (+ 721us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:57:04.749866 (+ 566us) server_negotiation.cc:1035] Waiting for connection context
0807 22:57:04.750477 (+ 611us) server_negotiation.cc:299] Negotiation successful
0807 22:57:04.750779 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32249,"thread_start_us":32046,"threads_started":1}
I20260807 22:57:04.753414 4788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:04.649332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:32777 (local address 127.25.94.1:41937)
0807 22:57:04.691028 (+ 41696us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:04.691049 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 22:57:04.695041 (+ 3992us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:57:04.696969 (+ 1928us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:57:04.696979 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:57:04.702021 (+ 5042us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:57:04.713386 (+ 11365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:04.713409 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:04.715795 (+ 2386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:04.715804 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:04.726513 (+ 10709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:04.726535 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:04.728421 (+ 1886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:04.728432 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:04.728894 (+ 462us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:04.730741 (+ 1847us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:57:04.730817 (+ 76us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:57:04.738568 (+ 7751us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:57:04.745908 (+ 7340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:57:04.745914 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:57:04.745918 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:57:04.747211 (+ 1293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:57:04.747852 (+ 641us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:57:04.747857 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:57:04.747860 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:57:04.748377 (+ 517us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:57:04.749968 (+ 1591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:57:04.749972 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:57:04.750160 (+ 188us) client_negotiation.cc:769] Sending connection context
0807 22:57:04.752036 (+ 1876us) client_negotiation.cc:240] Negotiation successful
0807 22:57:04.752372 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41491,"mutex_wait_us":4,"spinlock_wait_cycles":130816,"thread_start_us":29686,"threads_started":1}
I20260807 22:57:04.753927 4790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:04.647528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.60:40149 (local address 127.25.94.1:60465)
0807 22:57:04.689721 (+ 42193us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:04.689745 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0807 22:57:04.695710 (+ 5965us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:57:04.696991 (+ 1281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:57:04.696998 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:57:04.699251 (+ 2253us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:57:04.703240 (+ 3989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:04.703272 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:04.705502 (+ 2230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:04.705508 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:04.728795 (+ 23287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:04.728818 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:04.731139 (+ 2321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:04.731158 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:04.731565 (+ 407us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:04.732528 (+ 963us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:57:04.732554 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:57:04.738568 (+ 6014us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:57:04.745299 (+ 6731us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:57:04.745310 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:57:04.745331 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:57:04.746345 (+ 1014us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:57:04.747001 (+ 656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:57:04.747006 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:57:04.747010 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:57:04.747410 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:57:04.748230 (+ 820us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:57:04.748238 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:57:04.748743 (+ 505us) client_negotiation.cc:769] Sending connection context
0807 22:57:04.750702 (+ 1959us) client_negotiation.cc:240] Negotiation successful
0807 22:57:04.751326 (+ 624us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41921,"mutex_wait_us":496,"thread_start_us":33272,"threads_started":1}
I20260807 22:57:04.754186 4786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:04.645077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.1:38463 (local address 127.25.94.61:45205)
0807 22:57:04.682922 (+ 37845us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:04.682933 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:04.691389 (+ 8456us) server_negotiation.cc:407] Connection header received
0807 22:57:04.700728 (+ 9339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:04.700734 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:04.700893 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:04.701005 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:04.723939 (+ 22934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:04.725853 (+ 1914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:04.732534 (+ 6681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:04.735155 (+ 2621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:04.735412 (+ 257us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:04.742569 (+ 7157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:57:04.742578 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:57:04.742583 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:57:04.742628 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:57:04.749497 (+ 6869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:04.750438 (+ 941us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:04.750443 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:04.750447 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:04.750670 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:04.751341 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:04.751346 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:04.751350 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:04.751855 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:57:04.751993 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0807 22:57:04.753440 (+ 1447us) server_negotiation.cc:299] Negotiation successful
0807 22:57:04.753693 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37682,"thread_start_us":43266,"threads_started":1}
I20260807 22:57:04.755208 4796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:04.645980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.61:45205 (local address 127.25.94.1:38463)
0807 22:57:04.683927 (+ 37947us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:04.684001 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0807 22:57:04.700476 (+ 16475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:57:04.701452 (+ 976us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:57:04.701457 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:57:04.702295 (+ 838us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:57:04.723728 (+ 21433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:04.723753 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:04.726177 (+ 2424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:04.726187 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:04.730136 (+ 3949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:04.730155 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:04.737025 (+ 6870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:04.737034 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:04.737545 (+ 511us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:04.738359 (+ 814us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:57:04.738381 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:57:04.742149 (+ 3768us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:57:04.749668 (+ 7519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:57:04.749674 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:57:04.749681 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:57:04.750290 (+ 609us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:57:04.750816 (+ 526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:57:04.750821 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:57:04.750824 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:57:04.751185 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:57:04.752069 (+ 884us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:57:04.752073 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:57:04.752273 (+ 200us) client_negotiation.cc:769] Sending connection context
0807 22:57:04.752412 (+ 139us) client_negotiation.cc:240] Negotiation successful
0807 22:57:04.752848 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37285,"mutex_wait_us":1142,"thread_start_us":44442,"threads_started":1}
I20260807 22:57:04.755468 4780 heartbeater.cc:347] Connected to a master server at 127.25.94.60:40149
I20260807 22:57:04.755564 4783 heartbeater.cc:347] Connected to a master server at 127.25.94.62:32777
I20260807 22:57:04.755893 4780 heartbeater.cc:464] Registering TS with master...
I20260807 22:57:04.755914 4783 heartbeater.cc:464] Registering TS with master...
I20260807 22:57:04.756840 4784 heartbeater.cc:347] Connected to a master server at 127.25.94.61:45205
I20260807 22:57:04.757098 4784 heartbeater.cc:464] Registering TS with master...
I20260807 22:57:04.758425 4783 heartbeater.cc:511] Master 127.25.94.62:32777 requested a full tablet report, sending...
I20260807 22:57:04.758517 4780 heartbeater.cc:511] Master 127.25.94.60:40149 requested a full tablet report, sending...
I20260807 22:57:04.758658 4784 heartbeater.cc:511] Master 127.25.94.61:45205 requested a full tablet report, sending...
I20260807 22:57:04.761054 3206 ts_manager.cc:194] Re-registered known tserver with Master: 00fdb2f332f3437ea55e22ad5bf8ad14 (127.25.94.1:38459)
I20260807 22:57:04.761593 4157 ts_manager.cc:194] Registered new tserver with Master: 00fdb2f332f3437ea55e22ad5bf8ad14 (127.25.94.1:38459)
I20260807 22:57:04.762423 3206 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.25.94.1@KRBTEST.COM'} at 127.25.94.1:41937
I20260807 22:57:04.763985 4470 ts_manager.cc:194] Registered new tserver with Master: 00fdb2f332f3437ea55e22ad5bf8ad14 (127.25.94.1:38459)
I20260807 22:57:04.826200 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 3515
I20260807 22:57:04.849011 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.2:36449
--local_ip_for_outbound_sockets=127.25.94.2
--tserver_master_addrs=127.25.94.62:32777
--webserver_port=39967
--webserver_interface=127.25.94.2
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/krb5kdc/oryx_127.25.94.2.keytab
--principal=oryx/127.25.94.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.20:33185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.25.94.62:32777,127.25.94.61:45205
--tserver_master_addrs=127.25.94.62:32777,127.25.94.61:45205,127.25.94.60:40149 with env {KRB5CCNAME=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:57:05.198729 4815 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:57:05.199050 4815 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:57:05.199148 4815 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:57:05.207595 4815 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:57:05.207793 4815 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:57:05.208038 4815 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.2
I20260807 22:57:05.219209 4815 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:33185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/krb5kdc/oryx_127.25.94.2.keytab
--principal=oryx/127.25.94.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.2:36449
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.94.2
--webserver_port=39967
--tserver_master_addrs=127.25.94.62:32777,127.25.94.61:45205,127.25.94.60:40149
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.2
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 22:57:05.220680 4815 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:57:05.222122 4815 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:57:05.237756 4823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:57:05.237771 4821 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:57:05.237741 4827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:57:05.243212 4815 server_base.cc:1034] running on GCE node
I20260807 22:57:05.244170 4815 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:57:05.245412 4815 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:57:05.246634 4815 hybrid_clock.cc:648] HybridClock initialized: now 1786143425246561 us; error 67 us; skew 500 ppm
I20260807 22:57:05.247030 4815 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 22:57:05 dist-test-slave-z8x9 krb5kdc[3141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143425, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.94.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:57:05.250689 4815 init.cc:377] Logged in from keytab as oryx/127.25.94.2@KRBTEST.COM (short username oryx)
I20260807 22:57:05.252527 4815 webserver.cc:460] Webserver started at http://127.25.94.2:39967/ using document root <none> and password file <none>
I20260807 22:57:05.253494 4815 fs_manager.cc:359] Metadata directory not provided
I20260807 22:57:05.253628 4815 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:57:05.259719 4815 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260807 22:57:05.262698 4840 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:57:05.264302 4815 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 22:57:05.264506 4815 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/ts-1/wal
uuid: "c9f865cfe1654d8f88a210773ef5d9bd"
format_stamp: "Formatted at 2026-08-07 22:57:01 on dist-test-slave-z8x9"
server_key: "ee33525b0ade321cc82eb098fda049e4"
server_key_iv: "78921ba40816d9ff0bc85e50e8ab8070"
server_key_version: "encryptionkey@0"
I20260807 22:57:05.265813 4815 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:57:05.281517 4815 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:57:05.287127 4815 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:57:05.287457 4815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:57:05.288513 4815 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:57:05.290053 4815 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:57:05.290180 4815 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:57:05.290326 4815 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:57:05.290421 4815 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:57:05.334771 4815 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.2:36449
I20260807 22:57:05.334926 5029 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.2:36449 every 8 connection(s)
I20260807 22:57:05.336369 4815 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/ts-1/data/info.pb
I20260807 22:57:05.345607 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 4815
I20260807 22:57:05.346032 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 3719
Aug 07 22:57:05 dist-test-slave-z8x9 krb5kdc[3141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143419, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.25.94.3@KRBTEST.COM
Aug 07 22:57:05 dist-test-slave-z8x9 krb5kdc[3141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143425, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.94.2@KRBTEST.COM for oryx/127.25.94.62@KRBTEST.COM
Aug 07 22:57:05 dist-test-slave-z8x9 krb5kdc[3141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143425, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.94.2@KRBTEST.COM for oryx/127.25.94.61@KRBTEST.COM
Aug 07 22:57:05 dist-test-slave-z8x9 krb5kdc[3141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143425, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.94.2@KRBTEST.COM for oryx/127.25.94.60@KRBTEST.COM
I20260807 22:57:05.413625 5053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:05.348091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48470 (local address 127.25.94.3:37625)
0807 22:57:05.379120 (+ 31029us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:05.379136 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:05.379186 (+ 50us) server_negotiation.cc:407] Connection header received
0807 22:57:05.379307 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:05.379321 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:05.379851 (+ 530us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:05.380118 (+ 267us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:05.382610 (+ 2492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:05.383837 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:05.385620 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:05.386495 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:05.387398 (+ 903us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:05.400121 (+ 12723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:57:05.400146 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:57:05.400170 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:57:05.400252 (+ 82us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:57:05.406886 (+ 6634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:05.409703 (+ 2817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:05.409717 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:05.409723 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:05.410050 (+ 327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:05.410997 (+ 947us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:05.411003 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:05.411016 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:05.411748 (+ 732us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:57:05.412274 (+ 526us) server_negotiation.cc:1035] Waiting for connection context
0807 22:57:05.412659 (+ 385us) server_negotiation.cc:299] Negotiation successful
0807 22:57:05.413032 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30749,"thread_start_us":32369,"threads_started":1}
I20260807 22:57:05.415891 3910 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 22:57:05.420217 5040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:05.339993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.2:39417 (local address 127.25.94.60:40149)
0807 22:57:05.372429 (+ 32436us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:05.372439 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:05.384399 (+ 11960us) server_negotiation.cc:407] Connection header received
0807 22:57:05.388256 (+ 3857us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:05.388262 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:05.388352 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:05.388442 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:05.392814 (+ 4372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:05.394493 (+ 1679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:05.398827 (+ 4334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:05.399904 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:05.400015 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:05.409684 (+ 9669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:57:05.409693 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:57:05.409699 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:57:05.409744 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:57:05.414734 (+ 4990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:05.415833 (+ 1099us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:05.415840 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:05.415845 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:05.416145 (+ 300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:05.417053 (+ 908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:05.417061 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:05.417067 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:05.417554 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:57:05.417783 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0807 22:57:05.419298 (+ 1515us) server_negotiation.cc:299] Negotiation successful
0807 22:57:05.419562 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32213,"thread_start_us":32503,"threads_started":1}
I20260807 22:57:05.420995 5038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:05.339696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.2:35487 (local address 127.25.94.62:32777)
0807 22:57:05.373090 (+ 33394us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:05.373101 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:05.385535 (+ 12434us) server_negotiation.cc:407] Connection header received
0807 22:57:05.388190 (+ 2655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:05.388197 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:05.388325 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:05.388459 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:05.391208 (+ 2749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:05.392150 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:05.395831 (+ 3681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:05.396459 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:05.396566 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:05.404434 (+ 7868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:57:05.404454 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:57:05.404459 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:57:05.404538 (+ 79us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:57:05.409235 (+ 4697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:05.411106 (+ 1871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:05.411116 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:05.411121 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:05.412975 (+ 1854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:05.417315 (+ 4340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:05.417325 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:05.417331 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:05.417924 (+ 593us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:57:05.418140 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0807 22:57:05.419739 (+ 1599us) server_negotiation.cc:299] Negotiation successful
0807 22:57:05.420054 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33199,"thread_start_us":33427,"threads_started":1}
I20260807 22:57:05.422005 5048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:05.340369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:32777 (local address 127.25.94.2:35487)
0807 22:57:05.385354 (+ 44985us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:05.385376 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 22:57:05.387876 (+ 2500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:57:05.388732 (+ 856us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:57:05.388740 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:57:05.389966 (+ 1226us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:57:05.390974 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:05.390995 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:05.392324 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:05.392330 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:05.395635 (+ 3305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:05.395657 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:05.396766 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:05.396789 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:05.397317 (+ 528us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:05.398239 (+ 922us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:57:05.398450 (+ 211us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:57:05.403722 (+ 5272us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:57:05.409390 (+ 5668us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:57:05.409400 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:57:05.409423 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:57:05.410399 (+ 976us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:57:05.416533 (+ 6134us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:57:05.416537 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:57:05.416540 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:57:05.416801 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:57:05.418085 (+ 1284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:57:05.418090 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:57:05.418476 (+ 386us) client_negotiation.cc:769] Sending connection context
0807 22:57:05.418883 (+ 407us) client_negotiation.cc:240] Negotiation successful
0807 22:57:05.419570 (+ 687us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42881,"mutex_wait_us":42,"spinlock_wait_cycles":406016,"thread_start_us":46596,"threads_started":1}
I20260807 22:57:05.422330 5049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:05.340369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.60:40149 (local address 127.25.94.2:39417)
0807 22:57:05.384143 (+ 43774us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:05.384231 (+ 88us) client_negotiation.cc:174] Beginning negotiation
0807 22:57:05.387871 (+ 3640us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:57:05.388733 (+ 862us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:57:05.388741 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:57:05.390027 (+ 1286us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:57:05.392602 (+ 2575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:05.392618 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:05.394763 (+ 2145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:05.394771 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:05.398659 (+ 3888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:05.398670 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:05.400761 (+ 2091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:05.400768 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:05.401118 (+ 350us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:05.402331 (+ 1213us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:57:05.402363 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:57:05.409354 (+ 6991us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:57:05.414892 (+ 5538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:57:05.414896 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:57:05.414899 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:57:05.415656 (+ 757us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:57:05.416309 (+ 653us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:57:05.416318 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:57:05.416324 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:57:05.416875 (+ 551us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:57:05.417749 (+ 874us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:57:05.417759 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:57:05.418396 (+ 637us) client_negotiation.cc:769] Sending connection context
0807 22:57:05.418883 (+ 487us) client_negotiation.cc:240] Negotiation successful
0807 22:57:05.420884 (+ 2001us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43431,"mutex_wait_us":1612,"spinlock_wait_cycles":132352,"thread_start_us":45035,"threads_started":1}
I20260807 22:57:05.424391 5031 heartbeater.cc:347] Connected to a master server at 127.25.94.60:40149
I20260807 22:57:05.424922 5031 heartbeater.cc:464] Registering TS with master...
I20260807 22:57:05.426410 5045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:05.340691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.2:52605 (local address 127.25.94.61:45205)
0807 22:57:05.373624 (+ 32933us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:05.373633 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:05.387661 (+ 14028us) server_negotiation.cc:407] Connection header received
0807 22:57:05.388326 (+ 665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:05.388331 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:05.388450 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:05.388552 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:05.392817 (+ 4265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:05.394495 (+ 1678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:05.398827 (+ 4332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:05.399931 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:05.400022 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:05.408081 (+ 8059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:57:05.408090 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:57:05.408095 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:57:05.408160 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:57:05.416271 (+ 8111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:05.422484 (+ 6213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:05.422494 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:05.422499 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:05.422876 (+ 377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:05.424317 (+ 1441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:05.424321 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:05.424324 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:05.424792 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:57:05.424965 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0807 22:57:05.425445 (+ 480us) server_negotiation.cc:299] Negotiation successful
0807 22:57:05.425764 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32806,"thread_start_us":32701,"threads_started":1}
I20260807 22:57:05.426690 5043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:05.340369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.61:45205 (local address 127.25.94.2:52605)
0807 22:57:05.387445 (+ 47076us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:05.387465 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 22:57:05.387871 (+ 406us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:57:05.388733 (+ 862us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:57:05.388741 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:57:05.390366 (+ 1625us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:57:05.392610 (+ 2244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:05.392619 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:05.394744 (+ 2125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:05.394750 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:05.398621 (+ 3871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:05.398638 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:05.400698 (+ 2060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:05.400721 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:05.401118 (+ 397us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:05.402327 (+ 1209us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:57:05.402355 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:57:05.407775 (+ 5420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:57:05.420005 (+ 12230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:57:05.420011 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:57:05.420014 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:57:05.421916 (+ 1902us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:57:05.423376 (+ 1460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:57:05.423383 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:57:05.423387 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:57:05.423693 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:57:05.424922 (+ 1229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:57:05.424927 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:57:05.425129 (+ 202us) client_negotiation.cc:769] Sending connection context
0807 22:57:05.425311 (+ 182us) client_negotiation.cc:240] Negotiation successful
0807 22:57:05.425697 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46889,"mutex_wait_us":1574,"spinlock_wait_cycles":9984,"thread_start_us":44806,"threads_started":1}
I20260807 22:57:05.427039 5033 heartbeater.cc:347] Connected to a master server at 127.25.94.62:32777
I20260807 22:57:05.427275 5033 heartbeater.cc:464] Registering TS with master...
I20260807 22:57:05.426769 5031 heartbeater.cc:511] Master 127.25.94.60:40149 requested a full tablet report, sending...
I20260807 22:57:05.428004 5034 heartbeater.cc:347] Connected to a master server at 127.25.94.61:45205
I20260807 22:57:05.428246 5034 heartbeater.cc:464] Registering TS with master...
I20260807 22:57:05.428666 5033 heartbeater.cc:511] Master 127.25.94.62:32777 requested a full tablet report, sending...
I20260807 22:57:05.429754 5034 heartbeater.cc:511] Master 127.25.94.61:45205 requested a full tablet report, sending...
I20260807 22:57:05.431775 4470 ts_manager.cc:194] Registered new tserver with Master: c9f865cfe1654d8f88a210773ef5d9bd (127.25.94.2:36449)
I20260807 22:57:05.431915 4157 ts_manager.cc:194] Registered new tserver with Master: c9f865cfe1654d8f88a210773ef5d9bd (127.25.94.2:36449)
I20260807 22:57:05.432947 3206 ts_manager.cc:194] Re-registered known tserver with Master: c9f865cfe1654d8f88a210773ef5d9bd (127.25.94.2:36449)
I20260807 22:57:05.434504 3206 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.25.94.2@KRBTEST.COM'} at 127.25.94.2:35487
I20260807 22:57:05.521505 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 3719
I20260807 22:57:05.548715 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.3:37625
--local_ip_for_outbound_sockets=127.25.94.3
--tserver_master_addrs=127.25.94.62:32777
--webserver_port=40397
--webserver_interface=127.25.94.3
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/krb5kdc/oryx_127.25.94.3.keytab
--principal=oryx/127.25.94.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.20:33185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.25.94.62:32777,127.25.94.61:45205
--tserver_master_addrs=127.25.94.62:32777,127.25.94.61:45205,127.25.94.60:40149 with env {KRB5CCNAME=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 22:57:05.766263 4783 heartbeater.cc:503] Master 127.25.94.62:32777 was elected leader, sending a full tablet report...
W20260807 22:57:05.890503 5055 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:57:05.890793 5055 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:57:05.890956 5055 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:57:05.898698 5055 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:57:05.898835 5055 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:57:05.899013 5055 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.3
I20260807 22:57:05.909920 5055 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:33185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/krb5kdc/oryx_127.25.94.3.keytab
--principal=oryx/127.25.94.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.3:37625
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.94.3
--webserver_port=40397
--tserver_master_addrs=127.25.94.62:32777,127.25.94.61:45205,127.25.94.60:40149
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.3
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 22:57:05.911456 5055 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:57:05.912817 5055 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:57:05.926858 5064 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:57:05.927052 5070 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:57:05.927580 5063 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:57:05.928598 5055 server_base.cc:1034] running on GCE node
I20260807 22:57:05.929344 5055 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:57:05.930316 5055 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:57:05.931520 5055 hybrid_clock.cc:648] HybridClock initialized: now 1786143425931479 us; error 31 us; skew 500 ppm
I20260807 22:57:05.931846 5055 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 22:57:05 dist-test-slave-z8x9 krb5kdc[3141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143425, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.94.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:57:05.935343 5055 init.cc:377] Logged in from keytab as oryx/127.25.94.3@KRBTEST.COM (short username oryx)
I20260807 22:57:05.936872 5055 webserver.cc:460] Webserver started at http://127.25.94.3:40397/ using document root <none> and password file <none>
I20260807 22:57:05.937721 5055 fs_manager.cc:359] Metadata directory not provided
I20260807 22:57:05.937840 5055 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:57:05.943441 5055 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.005s
I20260807 22:57:05.946633 5081 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:57:05.948310 5055 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260807 22:57:05.948503 5055 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/ts-2/wal
uuid: "974ee5dfbcb449f8849a6e84a6fff395"
format_stamp: "Formatted at 2026-08-07 22:57:02 on dist-test-slave-z8x9"
server_key: "39bff9f91deddba698c9f750898a6aea"
server_key_iv: "49d298e22dd350608f3df02a620bc48d"
server_key_version: "encryptionkey@0"
I20260807 22:57:05.949594 5055 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:57:05.966956 5055 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:57:05.971918 5055 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:57:05.972244 5055 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:57:05.973417 5055 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:57:05.974867 5055 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:57:05.974959 5055 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:57:05.975123 5055 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:57:05.975234 5055 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:57:06.017872 5055 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.3:37625
I20260807 22:57:06.017971 5264 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.3:37625 every 8 connection(s)
I20260807 22:57:06.019392 5055 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/ts-2/data/info.pb
I20260807 22:57:06.024061 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 5055
Aug 07 22:57:06 dist-test-slave-z8x9 krb5kdc[3141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143425, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.94.3@KRBTEST.COM for oryx/127.25.94.61@KRBTEST.COM
Aug 07 22:57:06 dist-test-slave-z8x9 krb5kdc[3141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143425, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.94.3@KRBTEST.COM for oryx/127.25.94.60@KRBTEST.COM
Aug 07 22:57:06 dist-test-slave-z8x9 krb5kdc[3141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143425, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.94.3@KRBTEST.COM for oryx/127.25.94.62@KRBTEST.COM
I20260807 22:57:06.113652 5280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:06.024152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.3:44491 (local address 127.25.94.61:45205)
0807 22:57:06.057358 (+ 33206us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:06.057369 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:06.057406 (+ 37us) server_negotiation.cc:407] Connection header received
0807 22:57:06.057500 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:06.057508 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:06.057625 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:06.061161 (+ 3536us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:06.064286 (+ 3125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:06.065844 (+ 1558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:06.085495 (+ 19651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:06.086047 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:06.094585 (+ 8538us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:06.097849 (+ 3264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:57:06.097858 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:57:06.097865 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:57:06.097920 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:57:06.105559 (+ 7639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:06.107717 (+ 2158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:06.107743 (+ 26us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:06.107747 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:06.108101 (+ 354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:06.109051 (+ 950us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:06.109055 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:06.109059 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:06.109800 (+ 741us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:57:06.112369 (+ 2569us) server_negotiation.cc:1035] Waiting for connection context
0807 22:57:06.112517 (+ 148us) server_negotiation.cc:299] Negotiation successful
0807 22:57:06.112808 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32961,"thread_start_us":37038,"threads_started":1}
I20260807 22:57:06.115535 5282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:06.023175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.60:40149 (local address 127.25.94.3:34513)
0807 22:57:06.083975 (+ 60800us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:06.084004 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0807 22:57:06.084181 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:57:06.085957 (+ 1776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:57:06.085967 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:57:06.086815 (+ 848us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:57:06.087236 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:06.087253 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:06.089631 (+ 2378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:06.089639 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:06.091912 (+ 2273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:06.091934 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:06.093629 (+ 1695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:06.093639 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:06.093911 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:06.094754 (+ 843us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:57:06.094780 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:57:06.102490 (+ 7710us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:57:06.110174 (+ 7684us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:57:06.110180 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:57:06.110183 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:57:06.111280 (+ 1097us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:57:06.111933 (+ 653us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:57:06.111938 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:57:06.111941 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:57:06.112369 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:57:06.113467 (+ 1098us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:57:06.113472 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:57:06.113691 (+ 219us) client_negotiation.cc:769] Sending connection context
0807 22:57:06.113869 (+ 178us) client_negotiation.cc:240] Negotiation successful
0807 22:57:06.114247 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60571,"mutex_wait_us":2112,"thread_start_us":4321,"threads_started":1}
I20260807 22:57:06.116420 5276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:06.022571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.3:34513 (local address 127.25.94.60:40149)
0807 22:57:06.085081 (+ 62510us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:06.085093 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:06.085119 (+ 26us) server_negotiation.cc:407] Connection header received
0807 22:57:06.085223 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:06.085231 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:06.085337 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:06.085486 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:06.087544 (+ 2058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:06.089389 (+ 1845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:06.092169 (+ 2780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:06.093078 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:06.093260 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:06.102963 (+ 9703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:57:06.102973 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:57:06.102980 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:57:06.103036 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:57:06.110019 (+ 6983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:06.111497 (+ 1478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:06.111504 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:06.111507 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:06.111753 (+ 246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:06.112527 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:06.112532 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:06.112535 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:06.113070 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:57:06.113366 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0807 22:57:06.115450 (+ 2084us) server_negotiation.cc:299] Negotiation successful
0807 22:57:06.115811 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62299,"thread_start_us":62642,"threads_started":1}
I20260807 22:57:06.117869 5278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:06.024307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:32777 (local address 127.25.94.3:48981)
0807 22:57:06.083119 (+ 58812us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:06.083150 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0807 22:57:06.083359 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:57:06.083865 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:57:06.083870 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:57:06.084920 (+ 1050us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:57:06.085479 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:06.085495 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:06.087310 (+ 1815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:06.087315 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:06.088849 (+ 1534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:06.088865 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:06.094129 (+ 5264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:06.094149 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:06.097285 (+ 3136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:06.099192 (+ 1907us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:57:06.099228 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:57:06.103926 (+ 4698us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:57:06.111879 (+ 7953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:57:06.111885 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:57:06.111889 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:57:06.113222 (+ 1333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:57:06.115067 (+ 1845us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:57:06.115073 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:57:06.115092 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:57:06.115461 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:57:06.116361 (+ 900us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:57:06.116366 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:57:06.116574 (+ 208us) client_negotiation.cc:769] Sending connection context
0807 22:57:06.116710 (+ 136us) client_negotiation.cc:240] Negotiation successful
0807 22:57:06.117090 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58615,"thread_start_us":59199,"threads_started":1}
I20260807 22:57:06.117987 5277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:06.022881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.3:48981 (local address 127.25.94.62:32777)
0807 22:57:06.082459 (+ 59578us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:06.082470 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:06.083530 (+ 1060us) server_negotiation.cc:407] Connection header received
0807 22:57:06.083614 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:06.083620 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:06.083721 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:06.083836 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:06.086320 (+ 2484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:06.087139 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:06.088995 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:06.089628 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:06.089942 (+ 314us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:06.104228 (+ 14286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:57:06.104237 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:57:06.104241 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:57:06.104292 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:57:06.110751 (+ 6459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:06.113368 (+ 2617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:06.113375 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:06.113378 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:06.113640 (+ 262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:06.115623 (+ 1983us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:06.115628 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:06.115631 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:06.116194 (+ 563us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:57:06.116402 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0807 22:57:06.117034 (+ 632us) server_negotiation.cc:299] Negotiation successful
0807 22:57:06.117349 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59365,"thread_start_us":60337,"threads_started":1}
I20260807 22:57:06.118251 5283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:06.023175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.61:45205 (local address 127.25.94.3:44491)
0807 22:57:06.030139 (+ 6964us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:06.030189 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0807 22:57:06.034147 (+ 3958us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:57:06.058374 (+ 24227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:57:06.058397 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:57:06.059622 (+ 1225us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:57:06.060834 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:06.060873 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:06.066192 (+ 5319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:06.066204 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:06.074143 (+ 7939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:06.074180 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:06.089893 (+ 15713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:06.089902 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:06.090276 (+ 374us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:06.091289 (+ 1013us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:57:06.091337 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:57:06.096481 (+ 5144us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:57:06.106025 (+ 9544us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:57:06.106038 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:57:06.106073 (+ 35us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:57:06.107277 (+ 1204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:57:06.108275 (+ 998us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:57:06.108282 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:57:06.108286 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:57:06.108865 (+ 579us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:57:06.110037 (+ 1172us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:57:06.110047 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:57:06.110862 (+ 815us) client_negotiation.cc:769] Sending connection context
0807 22:57:06.111396 (+ 534us) client_negotiation.cc:240] Negotiation successful
0807 22:57:06.113556 (+ 2160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6624,"thread_start_us":59256,"threads_started":1}
I20260807 22:57:06.118630 5265 heartbeater.cc:347] Connected to a master server at 127.25.94.60:40149
I20260807 22:57:06.119148 5265 heartbeater.cc:464] Registering TS with master...
I20260807 22:57:06.119155 5267 heartbeater.cc:347] Connected to a master server at 127.25.94.62:32777
I20260807 22:57:06.119447 5267 heartbeater.cc:464] Registering TS with master...
I20260807 22:57:06.121471 5272 heartbeater.cc:347] Connected to a master server at 127.25.94.61:45205
I20260807 22:57:06.121762 5272 heartbeater.cc:464] Registering TS with master...
I20260807 22:57:06.123070 5265 heartbeater.cc:511] Master 127.25.94.60:40149 requested a full tablet report, sending...
I20260807 22:57:06.125013 5267 heartbeater.cc:511] Master 127.25.94.62:32777 requested a full tablet report, sending...
I20260807 22:57:06.125639 5272 heartbeater.cc:511] Master 127.25.94.61:45205 requested a full tablet report, sending...
I20260807 22:57:06.126165 4470 ts_manager.cc:194] Registered new tserver with Master: 974ee5dfbcb449f8849a6e84a6fff395 (127.25.94.3:37625)
I20260807 22:57:06.126570 3206 ts_manager.cc:194] Re-registered known tserver with Master: 974ee5dfbcb449f8849a6e84a6fff395 (127.25.94.3:37625)
I20260807 22:57:06.127789 4157 ts_manager.cc:194] Registered new tserver with Master: 974ee5dfbcb449f8849a6e84a6fff395 (127.25.94.3:37625)
I20260807 22:57:06.128518 3206 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.25.94.3@KRBTEST.COM'} at 127.25.94.3:48981
I20260807 22:57:06.154518 5291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:06.106864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41358 (local address 127.25.94.62:32777)
0807 22:57:06.113841 (+ 6977us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:06.113847 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:06.113862 (+ 15us) server_negotiation.cc:407] Connection header received
0807 22:57:06.113911 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:06.113918 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:06.113997 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:06.114098 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:06.125327 (+ 11229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:06.130350 (+ 5023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:06.133943 (+ 3593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:06.135288 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:06.135491 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:06.141665 (+ 6174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:57:06.141674 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:57:06.141679 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:57:06.141748 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:57:06.149305 (+ 7557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:06.150865 (+ 1560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:06.150872 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:06.150877 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:06.151140 (+ 263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:06.152007 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:06.152011 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:06.152013 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:06.152508 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:57:06.153001 (+ 493us) server_negotiation.cc:1035] Waiting for connection context
0807 22:57:06.153401 (+ 400us) server_negotiation.cc:299] Negotiation successful
0807 22:57:06.153904 (+ 503us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6204,"thread_start_us":6461,"threads_started":1}
I20260807 22:57:06.161567 5280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:06.119222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57856 (local address 127.25.94.61:45205)
0807 22:57:06.120698 (+ 1476us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:06.120703 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:06.130655 (+ 9952us) server_negotiation.cc:407] Connection header received
0807 22:57:06.130974 (+ 319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:06.130980 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:06.131161 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:06.131328 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:06.136940 (+ 5612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:06.139131 (+ 2191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:06.142407 (+ 3276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:06.143042 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:06.143220 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:06.148313 (+ 5093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:57:06.148320 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:57:06.148324 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:57:06.148362 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:57:06.156030 (+ 7668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:06.157720 (+ 1690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:06.157729 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:06.157735 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:06.158022 (+ 287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:06.158931 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:06.158937 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:06.158941 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:06.159852 (+ 911us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:57:06.160301 (+ 449us) server_negotiation.cc:1035] Waiting for connection context
0807 22:57:06.160782 (+ 481us) server_negotiation.cc:299] Negotiation successful
0807 22:57:06.161097 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1368}
I20260807 22:57:06.164357 5276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:06.120369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52392 (local address 127.25.94.60:40149)
0807 22:57:06.120809 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:06.120814 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:06.133553 (+ 12739us) server_negotiation.cc:407] Connection header received
0807 22:57:06.133919 (+ 366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:06.133924 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:06.134062 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:06.134125 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:06.139518 (+ 5393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:06.141426 (+ 1908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:06.144753 (+ 3327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:06.146020 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:06.146266 (+ 246us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:06.152267 (+ 6001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:57:06.152278 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:57:06.152288 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:57:06.152351 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:57:06.159508 (+ 7157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:06.160907 (+ 1399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:06.160924 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:06.160927 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:06.161240 (+ 313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:06.162337 (+ 1097us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:06.162342 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:06.162344 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:06.162798 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:57:06.163044 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0807 22:57:06.163743 (+ 699us) server_negotiation.cc:299] Negotiation successful
0807 22:57:06.163952 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349}
I20260807 22:57:06.160974 3206 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41358:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 22:57:06.175223 3206 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 22:57:06.243373 5312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:06.211787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.1:38459 (local address 127.0.0.1:45532)
0807 22:57:06.223505 (+ 11718us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:06.223542 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0807 22:57:06.223824 (+ 282us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:57:06.229315 (+ 5491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:57:06.229336 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:57:06.229360 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:57:06.229949 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:06.229961 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:06.233264 (+ 3303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:06.233270 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:06.237444 (+ 4174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:06.237473 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:06.242234 (+ 4761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:06.242241 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:06.242463 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:06.242495 (+ 32us) client_negotiation.cc:769] Sending connection context
0807 22:57:06.242591 (+ 96us) client_negotiation.cc:240] Negotiation successful
0807 22:57:06.242783 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11417,"thread_start_us":16754,"threads_started":1}
I20260807 22:57:06.243904 5313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:06.212872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48480 (local address 127.25.94.3:37625)
0807 22:57:06.229481 (+ 16609us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:06.229495 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:06.229603 (+ 108us) server_negotiation.cc:407] Connection header received
0807 22:57:06.229772 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:06.229786 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:06.230132 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:06.230494 (+ 362us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:57:06.232383 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:06.235451 (+ 3068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:06.239298 (+ 3847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:06.240857 (+ 1559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:06.241044 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:06.241430 (+ 386us) server_negotiation.cc:1035] Waiting for connection context
0807 22:57:06.242961 (+ 1531us) server_negotiation.cc:299] Negotiation successful
0807 22:57:06.243186 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15905,"thread_start_us":16260,"threads_started":1}
I20260807 22:57:06.244534 5307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:06.212099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.3:37625 (local address 127.0.0.1:48480)
0807 22:57:06.229103 (+ 17004us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:06.229123 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 22:57:06.229357 (+ 234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:57:06.230473 (+ 1116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:57:06.230478 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:57:06.230518 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:57:06.231933 (+ 1415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:06.231950 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:06.235714 (+ 3764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:06.235720 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:06.239083 (+ 3363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:06.239097 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:06.241208 (+ 2111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:06.241218 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:06.241956 (+ 738us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:06.242015 (+ 59us) client_negotiation.cc:769] Sending connection context
0807 22:57:06.242692 (+ 677us) client_negotiation.cc:240] Negotiation successful
0807 22:57:06.242802 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16799,"thread_start_us":11902,"threads_started":1}
I20260807 22:57:06.245810 5308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:06.212171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45532 (local address 127.25.94.1:38459)
0807 22:57:06.226395 (+ 14224us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:06.226409 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:06.226459 (+ 50us) server_negotiation.cc:407] Connection header received
0807 22:57:06.226615 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:06.226629 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:06.227078 (+ 449us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:06.227448 (+ 370us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:57:06.230152 (+ 2704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:06.232877 (+ 2725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:06.237664 (+ 4787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:06.240960 (+ 3296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:06.241147 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:06.241664 (+ 517us) server_negotiation.cc:1035] Waiting for connection context
0807 22:57:06.244934 (+ 3270us) server_negotiation.cc:299] Negotiation successful
0807 22:57:06.245140 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12878,"thread_start_us":13644,"threads_started":1}
I20260807 22:57:06.250438 4669 tablet_service.cc:1467] Processing CreateTablet for tablet ff264f6b773241538a9527691e8dea00 (DEFAULT_TABLE table=test-table [id=cfc80bf23e944e1b9d4b360ba3d5bed6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:57:06.250438 5153 tablet_service.cc:1467] Processing CreateTablet for tablet ff264f6b773241538a9527691e8dea00 (DEFAULT_TABLE table=test-table [id=cfc80bf23e944e1b9d4b360ba3d5bed6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:57:06.252593 5153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff264f6b773241538a9527691e8dea00. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:57:06.252739 4669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff264f6b773241538a9527691e8dea00. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:57:06.254760 5315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:06.231071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.2:36449 (local address 127.0.0.1:35046)
0807 22:57:06.242391 (+ 11320us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:06.242433 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0807 22:57:06.242627 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:57:06.243827 (+ 1200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:57:06.243834 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:57:06.243864 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:57:06.244997 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:06.245014 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:06.248210 (+ 3196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:06.248216 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:06.250612 (+ 2396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:06.250634 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:06.253467 (+ 2833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:06.253476 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:06.253764 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:06.253794 (+ 30us) client_negotiation.cc:769] Sending connection context
0807 22:57:06.253922 (+ 128us) client_negotiation.cc:240] Negotiation successful
0807 22:57:06.254100 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11166,"thread_start_us":10314,"threads_started":1}
I20260807 22:57:06.255244 5314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:06.225428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35046 (local address 127.25.94.2:36449)
0807 22:57:06.236280 (+ 10852us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:06.236293 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:06.242567 (+ 6274us) server_negotiation.cc:407] Connection header received
0807 22:57:06.242885 (+ 318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:06.242895 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:06.243195 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:06.243417 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:57:06.245256 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:06.248037 (+ 2781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:06.250819 (+ 2782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:06.252647 (+ 1828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:06.252850 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:06.253198 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0807 22:57:06.254110 (+ 912us) server_negotiation.cc:299] Negotiation successful
0807 22:57:06.254432 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10571,"thread_start_us":10749,"threads_started":1}
I20260807 22:57:06.259294 4924 tablet_service.cc:1467] Processing CreateTablet for tablet ff264f6b773241538a9527691e8dea00 (DEFAULT_TABLE table=test-table [id=cfc80bf23e944e1b9d4b360ba3d5bed6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:57:06.260794 4924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff264f6b773241538a9527691e8dea00. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:57:06.279902 5316 tablet_bootstrap.cc:492] T ff264f6b773241538a9527691e8dea00 P 00fdb2f332f3437ea55e22ad5bf8ad14: Bootstrap starting.
I20260807 22:57:06.282040 5317 tablet_bootstrap.cc:492] T ff264f6b773241538a9527691e8dea00 P 974ee5dfbcb449f8849a6e84a6fff395: Bootstrap starting.
I20260807 22:57:06.283952 5318 tablet_bootstrap.cc:492] T ff264f6b773241538a9527691e8dea00 P c9f865cfe1654d8f88a210773ef5d9bd: Bootstrap starting.
I20260807 22:57:06.287820 5316 tablet_bootstrap.cc:654] T ff264f6b773241538a9527691e8dea00 P 00fdb2f332f3437ea55e22ad5bf8ad14: Neither blocks nor log segments found. Creating new log.
I20260807 22:57:06.289661 5317 tablet_bootstrap.cc:654] T ff264f6b773241538a9527691e8dea00 P 974ee5dfbcb449f8849a6e84a6fff395: Neither blocks nor log segments found. Creating new log.
I20260807 22:57:06.290078 5316 log.cc:824] T ff264f6b773241538a9527691e8dea00 P 00fdb2f332f3437ea55e22ad5bf8ad14: Log is configured to *not* fsync() on all Append() calls
I20260807 22:57:06.291321 5318 tablet_bootstrap.cc:654] T ff264f6b773241538a9527691e8dea00 P c9f865cfe1654d8f88a210773ef5d9bd: Neither blocks nor log segments found. Creating new log.
I20260807 22:57:06.292771 5317 log.cc:824] T ff264f6b773241538a9527691e8dea00 P 974ee5dfbcb449f8849a6e84a6fff395: Log is configured to *not* fsync() on all Append() calls
I20260807 22:57:06.294137 5318 log.cc:824] T ff264f6b773241538a9527691e8dea00 P c9f865cfe1654d8f88a210773ef5d9bd: Log is configured to *not* fsync() on all Append() calls
I20260807 22:57:06.296000 5316 tablet_bootstrap.cc:492] T ff264f6b773241538a9527691e8dea00 P 00fdb2f332f3437ea55e22ad5bf8ad14: No bootstrap required, opened a new log
I20260807 22:57:06.296480 5316 ts_tablet_manager.cc:1397] T ff264f6b773241538a9527691e8dea00 P 00fdb2f332f3437ea55e22ad5bf8ad14: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.004s
I20260807 22:57:06.298559 5317 tablet_bootstrap.cc:492] T ff264f6b773241538a9527691e8dea00 P 974ee5dfbcb449f8849a6e84a6fff395: No bootstrap required, opened a new log
I20260807 22:57:06.298950 5318 tablet_bootstrap.cc:492] T ff264f6b773241538a9527691e8dea00 P c9f865cfe1654d8f88a210773ef5d9bd: No bootstrap required, opened a new log
I20260807 22:57:06.299095 5317 ts_tablet_manager.cc:1397] T ff264f6b773241538a9527691e8dea00 P 974ee5dfbcb449f8849a6e84a6fff395: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.011s
I20260807 22:57:06.299453 5318 ts_tablet_manager.cc:1397] T ff264f6b773241538a9527691e8dea00 P c9f865cfe1654d8f88a210773ef5d9bd: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.004s
I20260807 22:57:06.317879 5316 raft_consensus.cc:348] T ff264f6b773241538a9527691e8dea00 P 00fdb2f332f3437ea55e22ad5bf8ad14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fdb2f332f3437ea55e22ad5bf8ad14" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 38459 } } peers { permanent_uuid: "974ee5dfbcb449f8849a6e84a6fff395" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 37625 } } peers { permanent_uuid: "c9f865cfe1654d8f88a210773ef5d9bd" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 36449 } }
I20260807 22:57:06.318835 5316 raft_consensus.cc:374] T ff264f6b773241538a9527691e8dea00 P 00fdb2f332f3437ea55e22ad5bf8ad14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:57:06.319003 5316 raft_consensus.cc:729] T ff264f6b773241538a9527691e8dea00 P 00fdb2f332f3437ea55e22ad5bf8ad14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00fdb2f332f3437ea55e22ad5bf8ad14, State: Initialized, Role: FOLLOWER
I20260807 22:57:06.319206 5317 raft_consensus.cc:348] T ff264f6b773241538a9527691e8dea00 P 974ee5dfbcb449f8849a6e84a6fff395 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fdb2f332f3437ea55e22ad5bf8ad14" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 38459 } } peers { permanent_uuid: "974ee5dfbcb449f8849a6e84a6fff395" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 37625 } } peers { permanent_uuid: "c9f865cfe1654d8f88a210773ef5d9bd" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 36449 } }
I20260807 22:57:06.320185 5317 raft_consensus.cc:374] T ff264f6b773241538a9527691e8dea00 P 974ee5dfbcb449f8849a6e84a6fff395 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:57:06.320070 5316 consensus_queue.cc:260] T ff264f6b773241538a9527691e8dea00 P 00fdb2f332f3437ea55e22ad5bf8ad14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fdb2f332f3437ea55e22ad5bf8ad14" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 38459 } } peers { permanent_uuid: "974ee5dfbcb449f8849a6e84a6fff395" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 37625 } } peers { permanent_uuid: "c9f865cfe1654d8f88a210773ef5d9bd" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 36449 } }
I20260807 22:57:06.320420 5317 raft_consensus.cc:729] T ff264f6b773241538a9527691e8dea00 P 974ee5dfbcb449f8849a6e84a6fff395 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 974ee5dfbcb449f8849a6e84a6fff395, State: Initialized, Role: FOLLOWER
I20260807 22:57:06.321241 5317 consensus_queue.cc:260] T ff264f6b773241538a9527691e8dea00 P 974ee5dfbcb449f8849a6e84a6fff395 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fdb2f332f3437ea55e22ad5bf8ad14" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 38459 } } peers { permanent_uuid: "974ee5dfbcb449f8849a6e84a6fff395" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 37625 } } peers { permanent_uuid: "c9f865cfe1654d8f88a210773ef5d9bd" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 36449 } }
I20260807 22:57:06.321533 5318 raft_consensus.cc:348] T ff264f6b773241538a9527691e8dea00 P c9f865cfe1654d8f88a210773ef5d9bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fdb2f332f3437ea55e22ad5bf8ad14" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 38459 } } peers { permanent_uuid: "974ee5dfbcb449f8849a6e84a6fff395" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 37625 } } peers { permanent_uuid: "c9f865cfe1654d8f88a210773ef5d9bd" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 36449 } }
I20260807 22:57:06.322268 5318 raft_consensus.cc:374] T ff264f6b773241538a9527691e8dea00 P c9f865cfe1654d8f88a210773ef5d9bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:57:06.322516 5318 raft_consensus.cc:729] T ff264f6b773241538a9527691e8dea00 P c9f865cfe1654d8f88a210773ef5d9bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9f865cfe1654d8f88a210773ef5d9bd, State: Initialized, Role: FOLLOWER
I20260807 22:57:06.323645 5318 consensus_queue.cc:260] T ff264f6b773241538a9527691e8dea00 P c9f865cfe1654d8f88a210773ef5d9bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fdb2f332f3437ea55e22ad5bf8ad14" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 38459 } } peers { permanent_uuid: "974ee5dfbcb449f8849a6e84a6fff395" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 37625 } } peers { permanent_uuid: "c9f865cfe1654d8f88a210773ef5d9bd" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 36449 } }
I20260807 22:57:06.329056 5318 ts_tablet_manager.cc:1428] T ff264f6b773241538a9527691e8dea00 P c9f865cfe1654d8f88a210773ef5d9bd: Time spent starting tablet: real 0.029s user 0.018s sys 0.008s
W20260807 22:57:06.340539 5036 tablet.cc:2369] T ff264f6b773241538a9527691e8dea00 P c9f865cfe1654d8f88a210773ef5d9bd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 22:57:06.343276 5033 heartbeater.cc:503] Master 127.25.94.62:32777 was elected leader, sending a full tablet report...
I20260807 22:57:06.359316 5332 raft_consensus.cc:482] T ff264f6b773241538a9527691e8dea00 P c9f865cfe1654d8f88a210773ef5d9bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 22:57:06.360066 5332 raft_consensus.cc:504] T ff264f6b773241538a9527691e8dea00 P c9f865cfe1654d8f88a210773ef5d9bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fdb2f332f3437ea55e22ad5bf8ad14" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 38459 } } peers { permanent_uuid: "974ee5dfbcb449f8849a6e84a6fff395" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 37625 } } peers { permanent_uuid: "c9f865cfe1654d8f88a210773ef5d9bd" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 36449 } }
I20260807 22:57:06.363941 5316 ts_tablet_manager.cc:1428] T ff264f6b773241538a9527691e8dea00 P 00fdb2f332f3437ea55e22ad5bf8ad14: Time spent starting tablet: real 0.067s user 0.030s sys 0.037s
I20260807 22:57:06.377719 5317 ts_tablet_manager.cc:1428] T ff264f6b773241538a9527691e8dea00 P 974ee5dfbcb449f8849a6e84a6fff395: Time spent starting tablet: real 0.078s user 0.029s sys 0.049s
I20260807 22:57:06.378085 5332 leader_election.cc:288] T ff264f6b773241538a9527691e8dea00 P c9f865cfe1654d8f88a210773ef5d9bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00fdb2f332f3437ea55e22ad5bf8ad14 (127.25.94.1:38459), 974ee5dfbcb449f8849a6e84a6fff395 (127.25.94.3:37625)
I20260807 22:57:06.380640 5267 heartbeater.cc:503] Master 127.25.94.62:32777 was elected leader, sending a full tablet report...
I20260807 22:57:06.402065 5308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:06.386211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.2:45773 (local address 127.25.94.1:38459)
0807 22:57:06.386567 (+ 356us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:06.386574 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:06.391923 (+ 5349us) server_negotiation.cc:407] Connection header received
0807 22:57:06.392158 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:06.392164 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:06.392240 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:06.392369 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:57:06.393748 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:06.394500 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:06.399293 (+ 4793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:06.400657 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:06.400839 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:06.400982 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0807 22:57:06.401505 (+ 523us) server_negotiation.cc:299] Negotiation successful
0807 22:57:06.401650 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260807 22:57:06.402942 5313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:06.379033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.2:58137 (local address 127.25.94.3:37625)
0807 22:57:06.379331 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:06.379341 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:06.387925 (+ 8584us) server_negotiation.cc:407] Connection header received
0807 22:57:06.388231 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:06.388241 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:06.388343 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:06.388465 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:57:06.392309 (+ 3844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:06.393757 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:06.399396 (+ 5639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:06.401157 (+ 1761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:06.401449 (+ 292us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:06.401616 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0807 22:57:06.402474 (+ 858us) server_negotiation.cc:299] Negotiation successful
0807 22:57:06.402623 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260807 22:57:06.403319 5335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:06.385964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.1:38459 (local address 127.25.94.2:45773)
0807 22:57:06.391798 (+ 5834us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:06.391815 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 22:57:06.392010 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:57:06.392397 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:57:06.392402 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:57:06.392433 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:57:06.393453 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:06.393477 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:06.394639 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:06.394648 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:06.398991 (+ 4343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:06.399010 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:06.400975 (+ 1965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:06.400983 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:06.401286 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:06.401322 (+ 36us) client_negotiation.cc:769] Sending connection context
0807 22:57:06.402367 (+ 1045us) client_negotiation.cc:240] Negotiation successful
0807 22:57:06.402526 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5665,"mutex_wait_us":40,"thread_start_us":1198,"threads_started":1}
I20260807 22:57:06.404102 5337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:06.378666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.3:37625 (local address 127.25.94.2:58137)
0807 22:57:06.387747 (+ 9081us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:06.387806 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0807 22:57:06.388049 (+ 243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:57:06.388526 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:57:06.388531 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:57:06.388575 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:57:06.391860 (+ 3285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:06.391893 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:06.393965 (+ 2072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:06.393980 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:06.399103 (+ 5123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:06.399123 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:06.402008 (+ 2885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:06.402015 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:06.402281 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:06.402315 (+ 34us) client_negotiation.cc:769] Sending connection context
0807 22:57:06.403147 (+ 832us) client_negotiation.cc:240] Negotiation successful
0807 22:57:06.403314 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8830,"mutex_wait_us":49,"thread_start_us":12576,"threads_started":1}
I20260807 22:57:06.410511 4704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff264f6b773241538a9527691e8dea00" candidate_uuid: "c9f865cfe1654d8f88a210773ef5d9bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fdb2f332f3437ea55e22ad5bf8ad14" is_pre_election: true
I20260807 22:57:06.410658 5183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff264f6b773241538a9527691e8dea00" candidate_uuid: "c9f865cfe1654d8f88a210773ef5d9bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974ee5dfbcb449f8849a6e84a6fff395" is_pre_election: true
I20260807 22:57:06.411309 4704 raft_consensus.cc:2454] T ff264f6b773241538a9527691e8dea00 P 00fdb2f332f3437ea55e22ad5bf8ad14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9f865cfe1654d8f88a210773ef5d9bd in term 0.
I20260807 22:57:06.411461 5183 raft_consensus.cc:2454] T ff264f6b773241538a9527691e8dea00 P 974ee5dfbcb449f8849a6e84a6fff395 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9f865cfe1654d8f88a210773ef5d9bd in term 0.
W20260807 22:57:06.412250 4785 tablet.cc:2369] T ff264f6b773241538a9527691e8dea00 P 00fdb2f332f3437ea55e22ad5bf8ad14: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 22:57:06.412485 4851 leader_election.cc:302] T ff264f6b773241538a9527691e8dea00 P c9f865cfe1654d8f88a210773ef5d9bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fdb2f332f3437ea55e22ad5bf8ad14, c9f865cfe1654d8f88a210773ef5d9bd; no voters:
I20260807 22:57:06.413203 5333 raft_consensus.cc:2781] T ff264f6b773241538a9527691e8dea00 P c9f865cfe1654d8f88a210773ef5d9bd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 22:57:06.413582 5333 raft_consensus.cc:482] T ff264f6b773241538a9527691e8dea00 P c9f865cfe1654d8f88a210773ef5d9bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 22:57:06.414007 5333 raft_consensus.cc:3037] T ff264f6b773241538a9527691e8dea00 P c9f865cfe1654d8f88a210773ef5d9bd [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:57:06.418725 5333 raft_consensus.cc:504] T ff264f6b773241538a9527691e8dea00 P c9f865cfe1654d8f88a210773ef5d9bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fdb2f332f3437ea55e22ad5bf8ad14" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 38459 } } peers { permanent_uuid: "974ee5dfbcb449f8849a6e84a6fff395" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 37625 } } peers { permanent_uuid: "c9f865cfe1654d8f88a210773ef5d9bd" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 36449 } }
I20260807 22:57:06.419950 5333 leader_election.cc:288] T ff264f6b773241538a9527691e8dea00 P c9f865cfe1654d8f88a210773ef5d9bd [CANDIDATE]: Term 1 election: Requested vote from peers 00fdb2f332f3437ea55e22ad5bf8ad14 (127.25.94.1:38459), 974ee5dfbcb449f8849a6e84a6fff395 (127.25.94.3:37625)
I20260807 22:57:06.420315 4704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff264f6b773241538a9527691e8dea00" candidate_uuid: "c9f865cfe1654d8f88a210773ef5d9bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fdb2f332f3437ea55e22ad5bf8ad14"
I20260807 22:57:06.420583 4704 raft_consensus.cc:3037] T ff264f6b773241538a9527691e8dea00 P 00fdb2f332f3437ea55e22ad5bf8ad14 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:57:06.421099 5183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff264f6b773241538a9527691e8dea00" candidate_uuid: "c9f865cfe1654d8f88a210773ef5d9bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974ee5dfbcb449f8849a6e84a6fff395"
I20260807 22:57:06.421483 5183 raft_consensus.cc:3037] T ff264f6b773241538a9527691e8dea00 P 974ee5dfbcb449f8849a6e84a6fff395 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:57:06.424909 4704 raft_consensus.cc:2454] T ff264f6b773241538a9527691e8dea00 P 00fdb2f332f3437ea55e22ad5bf8ad14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9f865cfe1654d8f88a210773ef5d9bd in term 1.
I20260807 22:57:06.425648 4851 leader_election.cc:302] T ff264f6b773241538a9527691e8dea00 P c9f865cfe1654d8f88a210773ef5d9bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fdb2f332f3437ea55e22ad5bf8ad14, c9f865cfe1654d8f88a210773ef5d9bd; no voters:
I20260807 22:57:06.426162 5333 raft_consensus.cc:2781] T ff264f6b773241538a9527691e8dea00 P c9f865cfe1654d8f88a210773ef5d9bd [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:57:06.426415 5333 raft_consensus.cc:686] T ff264f6b773241538a9527691e8dea00 P c9f865cfe1654d8f88a210773ef5d9bd [term 1 LEADER]: Becoming Leader. State: Replica: c9f865cfe1654d8f88a210773ef5d9bd, State: Running, Role: LEADER
I20260807 22:57:06.427058 5183 raft_consensus.cc:2454] T ff264f6b773241538a9527691e8dea00 P 974ee5dfbcb449f8849a6e84a6fff395 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9f865cfe1654d8f88a210773ef5d9bd in term 1.
I20260807 22:57:06.428643 5333 consensus_queue.cc:237] T ff264f6b773241538a9527691e8dea00 P c9f865cfe1654d8f88a210773ef5d9bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fdb2f332f3437ea55e22ad5bf8ad14" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 38459 } } peers { permanent_uuid: "974ee5dfbcb449f8849a6e84a6fff395" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 37625 } } peers { permanent_uuid: "c9f865cfe1654d8f88a210773ef5d9bd" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 36449 } }
I20260807 22:57:06.435837 3206 catalog_manager.cc:5495] T ff264f6b773241538a9527691e8dea00 P c9f865cfe1654d8f88a210773ef5d9bd reported cstate change: term changed from 0 to 1, leader changed from <none> to c9f865cfe1654d8f88a210773ef5d9bd (127.25.94.2). New cstate: current_term: 1 leader_uuid: "c9f865cfe1654d8f88a210773ef5d9bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fdb2f332f3437ea55e22ad5bf8ad14" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 38459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "974ee5dfbcb449f8849a6e84a6fff395" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 37625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9f865cfe1654d8f88a210773ef5d9bd" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 36449 } health_report { overall_health: HEALTHY } } }
I20260807 22:57:06.479883 5314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:06.473323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35060 (local address 127.25.94.2:36449)
0807 22:57:06.473564 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:06.473570 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:06.473591 (+ 21us) server_negotiation.cc:407] Connection header received
0807 22:57:06.473801 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:06.473805 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:06.473903 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:06.474073 (+ 170us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 22:57:06.474665 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:06.475362 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:06.477317 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:06.477762 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:06.477912 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:06.478567 (+ 655us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 22:57:06.479106 (+ 539us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 22:57:06.479272 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0807 22:57:06.479355 (+ 83us) server_negotiation.cc:299] Negotiation successful
0807 22:57:06.479440 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260807 22:57:06.491459 5183 raft_consensus.cc:1261] T ff264f6b773241538a9527691e8dea00 P 974ee5dfbcb449f8849a6e84a6fff395 [term 1 FOLLOWER]: Refusing update from remote peer c9f865cfe1654d8f88a210773ef5d9bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 22:57:06.491707 4704 raft_consensus.cc:1261] T ff264f6b773241538a9527691e8dea00 P 00fdb2f332f3437ea55e22ad5bf8ad14 [term 1 FOLLOWER]: Refusing update from remote peer c9f865cfe1654d8f88a210773ef5d9bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 22:57:06.492791 5333 consensus_queue.cc:1035] T ff264f6b773241538a9527691e8dea00 P c9f865cfe1654d8f88a210773ef5d9bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "974ee5dfbcb449f8849a6e84a6fff395" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 37625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 22:57:06.493302 5332 consensus_queue.cc:1035] T ff264f6b773241538a9527691e8dea00 P c9f865cfe1654d8f88a210773ef5d9bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fdb2f332f3437ea55e22ad5bf8ad14" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 38459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 22:57:06.510277 5341 mvcc.cc:205] Tried to move back new op lower bound from 7316043474897084416 to 7316043474656972800. Current Snapshot: MvccSnapshot[applied={T|T < 7316043474897084416}]
I20260807 22:57:06.514333 5343 mvcc.cc:205] Tried to move back new op lower bound from 7316043474897084416 to 7316043474656972800. Current Snapshot: MvccSnapshot[applied={T|T < 7316043474897084416}]
I20260807 22:57:06.519493 5344 mvcc.cc:205] Tried to move back new op lower bound from 7316043474897084416 to 7316043474656972800. Current Snapshot: MvccSnapshot[applied={T|T < 7316043474897084416}]
I20260807 22:57:06.541884 3206 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41358:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 22:57:06.542212 3206 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41358:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 22:57:06.550808 3206 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9828cb5c031840cfac64207adc69ca5b: Sending DeleteTablet for 3 replicas of tablet ff264f6b773241538a9527691e8dea00
I20260807 22:57:06.553802 5153 tablet_service.cc:1514] Processing DeleteTablet for tablet ff264f6b773241538a9527691e8dea00 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:57:06 UTC) from {username='oryx', principal='oryx/127.25.94.62@KRBTEST.COM'} at 127.0.0.1:48480
I20260807 22:57:06.553757 4924 tablet_service.cc:1514] Processing DeleteTablet for tablet ff264f6b773241538a9527691e8dea00 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:57:06 UTC) from {username='oryx', principal='oryx/127.25.94.62@KRBTEST.COM'} at 127.0.0.1:35046
I20260807 22:57:06.554384 4669 tablet_service.cc:1514] Processing DeleteTablet for tablet ff264f6b773241538a9527691e8dea00 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:57:06 UTC) from {username='oryx', principal='oryx/127.25.94.62@KRBTEST.COM'} at 127.0.0.1:45532
I20260807 22:57:06.565672 5357 tablet_replica.cc:330] T ff264f6b773241538a9527691e8dea00 P 974ee5dfbcb449f8849a6e84a6fff395: stopping tablet replica
I20260807 22:57:06.566802 5357 raft_consensus.cc:2229] T ff264f6b773241538a9527691e8dea00 P 974ee5dfbcb449f8849a6e84a6fff395 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 22:57:06.568116 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 3161
I20260807 22:57:06.569767 3246 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 22:57:06.569731 5357 raft_consensus.cc:2258] T ff264f6b773241538a9527691e8dea00 P 974ee5dfbcb449f8849a6e84a6fff395 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 22:57:06.574335 5357 ts_tablet_manager.cc:1905] T ff264f6b773241538a9527691e8dea00 P 974ee5dfbcb449f8849a6e84a6fff395: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 22:57:06.575016 5362 tablet_replica.cc:330] T ff264f6b773241538a9527691e8dea00 P 00fdb2f332f3437ea55e22ad5bf8ad14: stopping tablet replica
I20260807 22:57:06.575618 5362 raft_consensus.cc:2229] T ff264f6b773241538a9527691e8dea00 P 00fdb2f332f3437ea55e22ad5bf8ad14 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 22:57:06.576116 5359 tablet_replica.cc:330] T ff264f6b773241538a9527691e8dea00 P c9f865cfe1654d8f88a210773ef5d9bd: stopping tablet replica
I20260807 22:57:06.576653 5362 raft_consensus.cc:2258] T ff264f6b773241538a9527691e8dea00 P 00fdb2f332f3437ea55e22ad5bf8ad14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 22:57:06.577312 5359 raft_consensus.cc:2229] T ff264f6b773241538a9527691e8dea00 P c9f865cfe1654d8f88a210773ef5d9bd [term 1 LEADER]: Raft consensus shutting down.
I20260807 22:57:06.580062 5362 ts_tablet_manager.cc:1905] T ff264f6b773241538a9527691e8dea00 P 00fdb2f332f3437ea55e22ad5bf8ad14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 22:57:06.580989 5359 raft_consensus.cc:2258] T ff264f6b773241538a9527691e8dea00 P c9f865cfe1654d8f88a210773ef5d9bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 22:57:06.594275 5359 ts_tablet_manager.cc:1905] T ff264f6b773241538a9527691e8dea00 P c9f865cfe1654d8f88a210773ef5d9bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 22:57:06.597849 5280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:06.557298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57860 (local address 127.25.94.61:45205)
0807 22:57:06.557548 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:06.557554 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:06.567812 (+ 10258us) server_negotiation.cc:407] Connection header received
0807 22:57:06.568065 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:06.568074 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:06.568208 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:06.568348 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:06.575518 (+ 7170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:06.577274 (+ 1756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:06.580003 (+ 2729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:06.580885 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:06.581022 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:06.586850 (+ 5828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:57:06.586858 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:57:06.586864 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:57:06.586967 (+ 103us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:57:06.592750 (+ 5783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:06.593952 (+ 1202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:06.593958 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:06.593961 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:06.594233 (+ 272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:06.594846 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:06.594851 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:06.594854 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:06.595402 (+ 548us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:57:06.595917 (+ 515us) server_negotiation.cc:1035] Waiting for connection context
0807 22:57:06.596804 (+ 887us) server_negotiation.cc:299] Negotiation successful
0807 22:57:06.597201 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260807 22:57:06.607450 5357 ts_tablet_manager.cc:1918] T ff264f6b773241538a9527691e8dea00 P 974ee5dfbcb449f8849a6e84a6fff395: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 22:57:06.608891 5357 log.cc:1196] T ff264f6b773241538a9527691e8dea00 P 974ee5dfbcb449f8849a6e84a6fff395: Deleting WAL directory at /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/ts-2/wal/wals/ff264f6b773241538a9527691e8dea00
I20260807 22:57:06.610316 5362 ts_tablet_manager.cc:1918] T ff264f6b773241538a9527691e8dea00 P 00fdb2f332f3437ea55e22ad5bf8ad14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 22:57:06.610687 5359 ts_tablet_manager.cc:1918] T ff264f6b773241538a9527691e8dea00 P c9f865cfe1654d8f88a210773ef5d9bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 22:57:06.611116 5357 ts_tablet_manager.cc:1939] T ff264f6b773241538a9527691e8dea00 P 974ee5dfbcb449f8849a6e84a6fff395: Deleting consensus metadata
I20260807 22:57:06.611214 5359 log.cc:1196] T ff264f6b773241538a9527691e8dea00 P c9f865cfe1654d8f88a210773ef5d9bd: Deleting WAL directory at /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/ts-1/wal/wals/ff264f6b773241538a9527691e8dea00
I20260807 22:57:06.611545 5362 log.cc:1196] T ff264f6b773241538a9527691e8dea00 P 00fdb2f332f3437ea55e22ad5bf8ad14: Deleting WAL directory at /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/ts-0/wal/wals/ff264f6b773241538a9527691e8dea00
I20260807 22:57:06.612279 5359 ts_tablet_manager.cc:1939] T ff264f6b773241538a9527691e8dea00 P c9f865cfe1654d8f88a210773ef5d9bd: Deleting consensus metadata
I20260807 22:57:06.613150 5362 ts_tablet_manager.cc:1939] T ff264f6b773241538a9527691e8dea00 P 00fdb2f332f3437ea55e22ad5bf8ad14: Deleting consensus metadata
I20260807 22:57:06.626338 5276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:06.565606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52394 (local address 127.25.94.60:40149)
0807 22:57:06.565891 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:06.565900 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:06.601200 (+ 35300us) server_negotiation.cc:407] Connection header received
0807 22:57:06.601709 (+ 509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:06.601716 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:06.601834 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:06.601928 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:06.604941 (+ 3013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:06.606528 (+ 1587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:06.609400 (+ 2872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:06.610375 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:06.610486 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:06.616665 (+ 6179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:57:06.616682 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:57:06.616688 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:57:06.616744 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:57:06.622270 (+ 5526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:06.623230 (+ 960us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:06.623236 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:06.623240 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:06.623451 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:06.623927 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:06.623944 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:06.623947 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:06.624547 (+ 600us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:57:06.625064 (+ 517us) server_negotiation.cc:1035] Waiting for connection context
0807 22:57:06.625267 (+ 203us) server_negotiation.cc:299] Negotiation successful
0807 22:57:06.625543 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260807 22:57:06.747377 3183 catalog_manager.cc:4841] TS c9f865cfe1654d8f88a210773ef5d9bd (127.25.94.2:36449): tablet ff264f6b773241538a9527691e8dea00 (table test-table [id=cfc80bf23e944e1b9d4b360ba3d5bed6]) successfully deleted
I20260807 22:57:06.748363 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 3161
I20260807 22:57:06.748332 3183 catalog_manager.cc:4841] TS 974ee5dfbcb449f8849a6e84a6fff395 (127.25.94.3:37625): tablet ff264f6b773241538a9527691e8dea00 (table test-table [id=cfc80bf23e944e1b9d4b360ba3d5bed6]) successfully deleted
I20260807 22:57:06.819233 5280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:06.798272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57874 (local address 127.25.94.61:45205)
0807 22:57:06.798515 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:06.798521 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:06.801851 (+ 3330us) server_negotiation.cc:407] Connection header received
0807 22:57:06.802455 (+ 604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:06.802462 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:06.802560 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:06.802641 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:06.805161 (+ 2520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:06.806491 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:06.808800 (+ 2309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:06.809626 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:06.809729 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:06.813061 (+ 3332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:57:06.813067 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:57:06.813070 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:57:06.813104 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:57:06.815960 (+ 2856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:06.816826 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:06.816834 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:06.816840 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:06.817015 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:06.817491 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:06.817496 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:06.817500 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:06.817915 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:57:06.818168 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0807 22:57:06.818610 (+ 442us) server_negotiation.cc:299] Negotiation successful
0807 22:57:06.818861 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260807 22:57:06.819517 5276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:06.798361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52410 (local address 127.25.94.60:40149)
0807 22:57:06.798579 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:06.798585 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:06.799875 (+ 1290us) server_negotiation.cc:407] Connection header received
0807 22:57:06.800214 (+ 339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:06.800224 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:06.800350 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:06.800446 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:06.803304 (+ 2858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:06.804585 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:06.807085 (+ 2500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:06.807769 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:06.807869 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:06.812378 (+ 4509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:57:06.812387 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:57:06.812393 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:57:06.812437 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:57:06.816290 (+ 3853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:06.817067 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:06.817073 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:06.817080 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:06.817294 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:06.817758 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:06.817763 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:06.817766 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:06.818149 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:57:06.818392 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0807 22:57:06.818762 (+ 370us) server_negotiation.cc:299] Negotiation successful
0807 22:57:06.819172 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260807 22:57:07.520131 5375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:07.519307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:32777 (local address 127.25.94.3:32835)
0807 22:57:07.519816 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:07.519937 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.62:32777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":271,"threads_started":1}
W20260807 22:57:07.520615 5267 heartbeater.cc:644] Failed to heartbeat to 127.25.94.62:32777 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.25.94.62:32777: connect: Connection refused (error 111)
I20260807 22:57:07.532884 5376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:07.531827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:32777 (local address 127.25.94.1:49545)
0807 22:57:07.532501 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:07.532655 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.62:32777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":452,"threads_started":1}
W20260807 22:57:07.533391 4783 heartbeater.cc:644] Failed to heartbeat to 127.25.94.62:32777 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.25.94.62:32777: connect: Connection refused (error 111)
I20260807 22:57:07.533399 5377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:07.532208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:32777 (local address 127.25.94.2:44963)
0807 22:57:07.532883 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:07.533242 (+ 359us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.62:32777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":450,"threads_started":1}
W20260807 22:57:07.533871 5033 heartbeater.cc:644] Failed to heartbeat to 127.25.94.62:32777 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.25.94.62:32777: connect: Connection refused (error 111)
I20260807 22:57:08.052045 5381 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5afdd40906094502ab5d825fbe1315a7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9828cb5c031840cfac64207adc69ca5b)
I20260807 22:57:08.052356 5381 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5afdd40906094502ab5d825fbe1315a7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9828cb5c031840cfac64207adc69ca5b" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 32777 } } peers { permanent_uuid: "5afdd40906094502ab5d825fbe1315a7" member_type: VOTER last_known_addr { host: "127.25.94.61" port: 45205 } attrs { promote: false } } peers { permanent_uuid: "6b8de2d663914806a8d5fc4fb0eda2b9" member_type: VOTER last_known_addr { host: "127.25.94.60" port: 40149 } attrs { promote: false } }
I20260807 22:57:08.052887 5382 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6b8de2d663914806a8d5fc4fb0eda2b9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9828cb5c031840cfac64207adc69ca5b)
I20260807 22:57:08.053130 5382 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6b8de2d663914806a8d5fc4fb0eda2b9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9828cb5c031840cfac64207adc69ca5b" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 32777 } } peers { permanent_uuid: "5afdd40906094502ab5d825fbe1315a7" member_type: VOTER last_known_addr { host: "127.25.94.61" port: 45205 } attrs { promote: false } } peers { permanent_uuid: "6b8de2d663914806a8d5fc4fb0eda2b9" member_type: VOTER last_known_addr { host: "127.25.94.60" port: 40149 } attrs { promote: false } }
I20260807 22:57:08.054155 5381 leader_election.cc:288] T 00000000000000000000000000000000 P 5afdd40906094502ab5d825fbe1315a7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9828cb5c031840cfac64207adc69ca5b (127.25.94.62:32777), 6b8de2d663914806a8d5fc4fb0eda2b9 (127.25.94.60:40149)
I20260807 22:57:08.055338 5383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:08.054235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:32777 (local address 127.0.0.1:41758)
0807 22:57:08.054769 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:08.054981 (+ 212us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.62:32777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":279,"threads_started":1}
W20260807 22:57:08.056366 4135 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.94.62:32777: connect: Connection refused (error 111)
I20260807 22:57:08.058017 5389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:08.056573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:32777 (local address 127.0.0.1:41750)
0807 22:57:08.057432 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:08.057670 (+ 238us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.62:32777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":702,"thread_start_us":781,"threads_started":1}
W20260807 22:57:08.058420 4448 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.94.62:32777: connect: Connection refused (error 111)
I20260807 22:57:08.065487 5382 leader_election.cc:288] T 00000000000000000000000000000000 P 6b8de2d663914806a8d5fc4fb0eda2b9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9828cb5c031840cfac64207adc69ca5b (127.25.94.62:32777), 5afdd40906094502ab5d825fbe1315a7 (127.25.94.61:45205)
I20260807 22:57:08.066584 5389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:08.066043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:32777 (local address 127.0.0.1:41766)
0807 22:57:08.066263 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:08.066384 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.62:32777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260807 22:57:08.067081 4448 leader_election.cc:334] T 00000000000000000000000000000000 P 6b8de2d663914806a8d5fc4fb0eda2b9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9828cb5c031840cfac64207adc69ca5b (127.25.94.62:32777): Network error: Client connection negotiation failed: client connection to 127.25.94.62:32777: connect: Connection refused (error 111)
I20260807 22:57:08.068125 5385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:08.054284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.61:45205 (local address 127.0.0.1:57886)
0807 22:57:08.057203 (+ 2919us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:08.057228 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0807 22:57:08.057620 (+ 392us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:57:08.058300 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:57:08.058308 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:57:08.058341 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:57:08.058765 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:08.058780 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:08.061004 (+ 2224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:08.061012 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:08.064000 (+ 2988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:08.064021 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:08.066958 (+ 2937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:08.066967 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:08.067327 (+ 360us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:08.067373 (+ 46us) client_negotiation.cc:769] Sending connection context
0807 22:57:08.067521 (+ 148us) client_negotiation.cc:240] Negotiation successful
0807 22:57:08.067669 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2712,"thread_start_us":2507,"threads_started":1}
I20260807 22:57:08.068591 5386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:08.054766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57886 (local address 127.25.94.61:45205)
0807 22:57:08.056402 (+ 1636us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:08.056408 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:08.057331 (+ 923us) server_negotiation.cc:407] Connection header received
0807 22:57:08.057817 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:08.057826 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:08.057933 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:08.058017 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:57:08.059037 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:08.060747 (+ 1710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:08.064605 (+ 3858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:08.066453 (+ 1848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:08.066586 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:08.066732 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0807 22:57:08.067607 (+ 875us) server_negotiation.cc:299] Negotiation successful
0807 22:57:08.067824 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1506,"thread_start_us":1341,"threads_started":1}
I20260807 22:57:08.069603 4175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6b8de2d663914806a8d5fc4fb0eda2b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "5afdd40906094502ab5d825fbe1315a7" is_pre_election: true
I20260807 22:57:08.069972 4175 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5afdd40906094502ab5d825fbe1315a7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b8de2d663914806a8d5fc4fb0eda2b9 in term 1.
I20260807 22:57:08.070747 4449 leader_election.cc:302] T 00000000000000000000000000000000 P 6b8de2d663914806a8d5fc4fb0eda2b9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5afdd40906094502ab5d825fbe1315a7, 6b8de2d663914806a8d5fc4fb0eda2b9; no voters: 9828cb5c031840cfac64207adc69ca5b
I20260807 22:57:08.071036 5384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:08.070403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:32777 (local address 127.0.0.1:41776)
0807 22:57:08.070642 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:08.070804 (+ 162us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.62:32777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260807 22:57:08.071372 5382 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6b8de2d663914806a8d5fc4fb0eda2b9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 22:57:08.071527 5382 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6b8de2d663914806a8d5fc4fb0eda2b9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9828cb5c031840cfac64207adc69ca5b)
I20260807 22:57:08.071663 5382 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6b8de2d663914806a8d5fc4fb0eda2b9 [term 1 FOLLOWER]: Advancing to term 2
W20260807 22:57:08.071610 4135 leader_election.cc:334] T 00000000000000000000000000000000 P 5afdd40906094502ab5d825fbe1315a7 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9828cb5c031840cfac64207adc69ca5b (127.25.94.62:32777): Network error: Client connection negotiation failed: client connection to 127.25.94.62:32777: connect: Connection refused (error 111)
I20260807 22:57:08.075053 5383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:08.054309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.60:40149 (local address 127.0.0.1:52422)
0807 22:57:08.055586 (+ 1277us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:08.055609 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0807 22:57:08.055776 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:57:08.069371 (+ 13595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:57:08.069376 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:57:08.069410 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:57:08.069736 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:08.069748 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:08.071412 (+ 1664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:08.071419 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:08.072987 (+ 1568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:08.073000 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:08.074256 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:08.074263 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:08.074447 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:08.074471 (+ 24us) client_negotiation.cc:769] Sending connection context
0807 22:57:08.074587 (+ 116us) client_negotiation.cc:240] Negotiation successful
0807 22:57:08.074730 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1157,"thread_start_us":10007,"threads_started":1}
I20260807 22:57:08.075127 5388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:08.054915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52422 (local address 127.25.94.60:40149)
0807 22:57:08.068962 (+ 14047us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:08.068967 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:08.068983 (+ 16us) server_negotiation.cc:407] Connection header received
0807 22:57:08.069042 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:08.069047 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:08.069124 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:08.069898 (+ 774us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:57:08.070117 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:08.071267 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:08.073185 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:08.074062 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:08.074211 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:08.074339 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0807 22:57:08.074611 (+ 272us) server_negotiation.cc:299] Negotiation successful
0807 22:57:08.074743 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13898,"thread_start_us":13741,"threads_started":1}
I20260807 22:57:08.075054 5382 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6b8de2d663914806a8d5fc4fb0eda2b9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9828cb5c031840cfac64207adc69ca5b" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 32777 } } peers { permanent_uuid: "5afdd40906094502ab5d825fbe1315a7" member_type: VOTER last_known_addr { host: "127.25.94.61" port: 45205 } attrs { promote: false } } peers { permanent_uuid: "6b8de2d663914806a8d5fc4fb0eda2b9" member_type: VOTER last_known_addr { host: "127.25.94.60" port: 40149 } attrs { promote: false } }
I20260807 22:57:08.075834 4487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5afdd40906094502ab5d825fbe1315a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "6b8de2d663914806a8d5fc4fb0eda2b9" is_pre_election: true
I20260807 22:57:08.076272 4487 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 6b8de2d663914806a8d5fc4fb0eda2b9 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5afdd40906094502ab5d825fbe1315a7 in current term 2: Already voted for candidate 6b8de2d663914806a8d5fc4fb0eda2b9 in this term.
I20260807 22:57:08.076275 5382 leader_election.cc:288] T 00000000000000000000000000000000 P 6b8de2d663914806a8d5fc4fb0eda2b9 [CANDIDATE]: Term 2 election: Requested vote from peers 9828cb5c031840cfac64207adc69ca5b (127.25.94.62:32777), 5afdd40906094502ab5d825fbe1315a7 (127.25.94.61:45205)
I20260807 22:57:08.076655 5385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:08.076257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:32777 (local address 127.0.0.1:41792)
0807 22:57:08.076411 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:08.076502 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.62:32777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260807 22:57:08.076691 4175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6b8de2d663914806a8d5fc4fb0eda2b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "5afdd40906094502ab5d825fbe1315a7"
I20260807 22:57:08.076927 4175 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5afdd40906094502ab5d825fbe1315a7 [term 1 FOLLOWER]: Advancing to term 2
I20260807 22:57:08.077121 4138 leader_election.cc:302] T 00000000000000000000000000000000 P 5afdd40906094502ab5d825fbe1315a7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5afdd40906094502ab5d825fbe1315a7; no voters: 6b8de2d663914806a8d5fc4fb0eda2b9, 9828cb5c031840cfac64207adc69ca5b
I20260807 22:57:08.078076 5385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:08.077679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:32777 (local address 127.0.0.1:41794)
0807 22:57:08.077807 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:08.077926 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.62:32777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260807 22:57:08.078336 4448 leader_election.cc:334] T 00000000000000000000000000000000 P 6b8de2d663914806a8d5fc4fb0eda2b9 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 9828cb5c031840cfac64207adc69ca5b (127.25.94.62:32777): Network error: Client connection negotiation failed: client connection to 127.25.94.62:32777: connect: Connection refused (error 111)
I20260807 22:57:08.079726 4175 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5afdd40906094502ab5d825fbe1315a7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b8de2d663914806a8d5fc4fb0eda2b9 in term 2.
I20260807 22:57:08.079921 5381 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 5afdd40906094502ab5d825fbe1315a7 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 22:57:08.080168 4449 leader_election.cc:302] T 00000000000000000000000000000000 P 6b8de2d663914806a8d5fc4fb0eda2b9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5afdd40906094502ab5d825fbe1315a7, 6b8de2d663914806a8d5fc4fb0eda2b9; no voters: 9828cb5c031840cfac64207adc69ca5b
I20260807 22:57:08.080665 5382 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6b8de2d663914806a8d5fc4fb0eda2b9 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 22:57:08.081123 5382 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6b8de2d663914806a8d5fc4fb0eda2b9 [term 2 LEADER]: Becoming Leader. State: Replica: 6b8de2d663914806a8d5fc4fb0eda2b9, State: Running, Role: LEADER
I20260807 22:57:08.081557 5382 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6b8de2d663914806a8d5fc4fb0eda2b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9828cb5c031840cfac64207adc69ca5b" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 32777 } } peers { permanent_uuid: "5afdd40906094502ab5d825fbe1315a7" member_type: VOTER last_known_addr { host: "127.25.94.61" port: 45205 } attrs { promote: false } } peers { permanent_uuid: "6b8de2d663914806a8d5fc4fb0eda2b9" member_type: VOTER last_known_addr { host: "127.25.94.60" port: 40149 } attrs { promote: false } }
I20260807 22:57:08.083611 5393 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b8de2d663914806a8d5fc4fb0eda2b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6b8de2d663914806a8d5fc4fb0eda2b9. Latest consensus state: current_term: 2 leader_uuid: "6b8de2d663914806a8d5fc4fb0eda2b9" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9828cb5c031840cfac64207adc69ca5b" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 32777 } } peers { permanent_uuid: "5afdd40906094502ab5d825fbe1315a7" member_type: VOTER last_known_addr { host: "127.25.94.61" port: 45205 } attrs { promote: false } } peers { permanent_uuid: "6b8de2d663914806a8d5fc4fb0eda2b9" member_type: VOTER last_known_addr { host: "127.25.94.60" port: 40149 } attrs { promote: false } } }
I20260807 22:57:08.084038 5393 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b8de2d663914806a8d5fc4fb0eda2b9 [sys.catalog]: This master's current role is: LEADER
I20260807 22:57:08.084730 5395 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 22:57:08.088763 5395 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 22:57:08.089712 5395 catalog_manager.cc:1259] Loaded cluster ID: b39391c33b034542a865d23e1ca84d55
I20260807 22:57:08.089792 5395 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 22:57:08.090974 5395 catalog_manager.cc:1504] Loading token signing keys...
I20260807 22:57:08.091792 5395 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 6b8de2d663914806a8d5fc4fb0eda2b9: Loaded TSK: 0
I20260807 22:57:08.092698 5395 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 22:57:08.103477 4470 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52410:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 22:57:08.111148 4470 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 22:57:08.115420 5385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:08.115017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:32777 (local address 127.0.0.1:41826)
0807 22:57:08.115187 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:08.115280 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.62:32777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260807 22:57:08.115592 4175 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5afdd40906094502ab5d825fbe1315a7 [term 2 FOLLOWER]: Refusing update from remote peer 6b8de2d663914806a8d5fc4fb0eda2b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260807 22:57:08.116422 5393 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6b8de2d663914806a8d5fc4fb0eda2b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5afdd40906094502ab5d825fbe1315a7" member_type: VOTER last_known_addr { host: "127.25.94.61" port: 45205 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260807 22:57:08.116632 5385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:08.116137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:32777 (local address 127.0.0.1:41832)
0807 22:57:08.116394 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:08.116485 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.62:32777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
W20260807 22:57:08.116815 4448 consensus_peers.cc:487] T 00000000000000000000000000000000 P 6b8de2d663914806a8d5fc4fb0eda2b9 -> Peer 9828cb5c031840cfac64207adc69ca5b (127.25.94.62:32777): Couldn't send request to peer 9828cb5c031840cfac64207adc69ca5b. Status: Network error: Client connection negotiation failed: client connection to 127.25.94.62:32777: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260807 22:57:08.119608 5381 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5afdd40906094502ab5d825fbe1315a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6b8de2d663914806a8d5fc4fb0eda2b9. Latest consensus state: current_term: 2 leader_uuid: "6b8de2d663914806a8d5fc4fb0eda2b9" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9828cb5c031840cfac64207adc69ca5b" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 32777 } } peers { permanent_uuid: "5afdd40906094502ab5d825fbe1315a7" member_type: VOTER last_known_addr { host: "127.25.94.61" port: 45205 } attrs { promote: false } } peers { permanent_uuid: "6b8de2d663914806a8d5fc4fb0eda2b9" member_type: VOTER last_known_addr { host: "127.25.94.60" port: 40149 } attrs { promote: false } } }
I20260807 22:57:08.119884 5381 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5afdd40906094502ab5d825fbe1315a7 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 22:57:08.121776 5382 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b8de2d663914806a8d5fc4fb0eda2b9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "6b8de2d663914806a8d5fc4fb0eda2b9" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9828cb5c031840cfac64207adc69ca5b" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 32777 } } peers { permanent_uuid: "5afdd40906094502ab5d825fbe1315a7" member_type: VOTER last_known_addr { host: "127.25.94.61" port: 45205 } attrs { promote: false } } peers { permanent_uuid: "6b8de2d663914806a8d5fc4fb0eda2b9" member_type: VOTER last_known_addr { host: "127.25.94.60" port: 40149 } attrs { promote: false } } }
I20260807 22:57:08.122201 5382 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b8de2d663914806a8d5fc4fb0eda2b9 [sys.catalog]: This master's current role is: LEADER
I20260807 22:57:08.122535 5381 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5afdd40906094502ab5d825fbe1315a7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "6b8de2d663914806a8d5fc4fb0eda2b9" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9828cb5c031840cfac64207adc69ca5b" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 32777 } } peers { permanent_uuid: "5afdd40906094502ab5d825fbe1315a7" member_type: VOTER last_known_addr { host: "127.25.94.61" port: 45205 } attrs { promote: false } } peers { permanent_uuid: "6b8de2d663914806a8d5fc4fb0eda2b9" member_type: VOTER last_known_addr { host: "127.25.94.60" port: 40149 } attrs { promote: false } } }
I20260807 22:57:08.122848 5381 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5afdd40906094502ab5d825fbe1315a7 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 22:57:08.123246 5399 mvcc.cc:205] Tried to move back new op lower bound from 7316043481554411520 to 7316043481425936384. Current Snapshot: MvccSnapshot[applied={T|T < 7316043481554411520 or (T in {7316043481554411520})}]
I20260807 22:57:08.146572 5389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:08.132528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.3:37625 (local address 127.0.0.1:48488)
0807 22:57:08.132676 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:08.132689 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0807 22:57:08.132799 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:57:08.139068 (+ 6269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:57:08.139078 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:57:08.139104 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:57:08.139597 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:08.139612 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:08.141060 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:08.141068 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:08.143515 (+ 2447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:08.143530 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:08.145705 (+ 2175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:08.145713 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:08.145902 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:08.145953 (+ 51us) client_negotiation.cc:769] Sending connection context
0807 22:57:08.146028 (+ 75us) client_negotiation.cc:240] Negotiation successful
0807 22:57:08.146198 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":126}
I20260807 22:57:08.147002 5385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:08.132162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.2:36449 (local address 127.0.0.1:35068)
0807 22:57:08.132374 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:08.132396 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 22:57:08.132522 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:57:08.138313 (+ 5791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:57:08.138318 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:57:08.138346 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:57:08.138571 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:08.138585 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:08.141154 (+ 2569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:08.141161 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:08.143759 (+ 2598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:08.143777 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:08.146071 (+ 2294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:08.146077 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:08.146291 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:08.146324 (+ 33us) client_negotiation.cc:769] Sending connection context
0807 22:57:08.146410 (+ 86us) client_negotiation.cc:240] Negotiation successful
0807 22:57:08.146542 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81,"mutex_wait_us":151}
I20260807 22:57:08.147485 5405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:08.132949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48488 (local address 127.25.94.3:37625)
0807 22:57:08.138757 (+ 5808us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:08.138763 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:08.138791 (+ 28us) server_negotiation.cc:407] Connection header received
0807 22:57:08.138849 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:08.138853 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:08.138924 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:08.139048 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:57:08.139839 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:08.140872 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:08.143799 (+ 2927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:08.145110 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:08.146872 (+ 1762us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:08.146987 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0807 22:57:08.147043 (+ 56us) server_negotiation.cc:299] Negotiation successful
0807 22:57:08.147207 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5663,"thread_start_us":314,"threads_started":1}
I20260807 22:57:08.147840 5404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:08.132340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35068 (local address 127.25.94.2:36449)
0807 22:57:08.137929 (+ 5589us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:08.137959 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:08.137985 (+ 26us) server_negotiation.cc:407] Connection header received
0807 22:57:08.138085 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:08.138090 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:08.138168 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:08.138290 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:57:08.138863 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:08.140002 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:08.144178 (+ 4176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:08.145857 (+ 1679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:08.146704 (+ 847us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:08.147366 (+ 662us) server_negotiation.cc:1035] Waiting for connection context
0807 22:57:08.147413 (+ 47us) server_negotiation.cc:299] Negotiation successful
0807 22:57:08.147521 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5467,"thread_start_us":5360,"threads_started":1}
I20260807 22:57:08.148341 5153 tablet_service.cc:1467] Processing CreateTablet for tablet b788452af6154bfb985175c9ab2de563 (DEFAULT_TABLE table=test-table [id=3a526c6fe3604845b77f996351bc3b89]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:57:08.148548 4924 tablet_service.cc:1467] Processing CreateTablet for tablet b788452af6154bfb985175c9ab2de563 (DEFAULT_TABLE table=test-table [id=3a526c6fe3604845b77f996351bc3b89]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:57:08.148764 5153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b788452af6154bfb985175c9ab2de563. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:57:08.148944 4924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b788452af6154bfb985175c9ab2de563. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:57:08.150417 5406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:08.133057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.1:38459 (local address 127.0.0.1:45536)
0807 22:57:08.133987 (+ 930us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:08.134004 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 22:57:08.134107 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:57:08.140167 (+ 6060us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:57:08.140174 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:57:08.140202 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:57:08.140926 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:08.140947 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:08.143505 (+ 2558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:08.143510 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:08.146289 (+ 2779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:08.146302 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:08.149503 (+ 3201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:08.149510 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:08.149714 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:08.149753 (+ 39us) client_negotiation.cc:769] Sending connection context
0807 22:57:08.149862 (+ 109us) client_negotiation.cc:240] Negotiation successful
0807 22:57:08.150010 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":807,"mutex_wait_us":3,"thread_start_us":630,"threads_started":1}
I20260807 22:57:08.150417 5407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:08.133465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45536 (local address 127.25.94.1:38459)
0807 22:57:08.139785 (+ 6320us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:08.139803 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:08.139821 (+ 18us) server_negotiation.cc:407] Connection header received
0807 22:57:08.139896 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:08.139903 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:08.139992 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:08.140106 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:57:08.141129 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:08.142170 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:08.146740 (+ 4570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:08.148000 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:08.148105 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:08.148188 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0807 22:57:08.149935 (+ 1747us) server_negotiation.cc:299] Negotiation successful
0807 22:57:08.150105 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6186,"thread_start_us":6039,"threads_started":1}
I20260807 22:57:08.152128 4669 tablet_service.cc:1467] Processing CreateTablet for tablet b788452af6154bfb985175c9ab2de563 (DEFAULT_TABLE table=test-table [id=3a526c6fe3604845b77f996351bc3b89]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:57:08.152627 4669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b788452af6154bfb985175c9ab2de563. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:57:08.155324 5408 tablet_bootstrap.cc:492] T b788452af6154bfb985175c9ab2de563 P 974ee5dfbcb449f8849a6e84a6fff395: Bootstrap starting.
I20260807 22:57:08.157410 5408 tablet_bootstrap.cc:654] T b788452af6154bfb985175c9ab2de563 P 974ee5dfbcb449f8849a6e84a6fff395: Neither blocks nor log segments found. Creating new log.
I20260807 22:57:08.158841 5409 tablet_bootstrap.cc:492] T b788452af6154bfb985175c9ab2de563 P c9f865cfe1654d8f88a210773ef5d9bd: Bootstrap starting.
I20260807 22:57:08.160663 5408 tablet_bootstrap.cc:492] T b788452af6154bfb985175c9ab2de563 P 974ee5dfbcb449f8849a6e84a6fff395: No bootstrap required, opened a new log
I20260807 22:57:08.160872 5408 ts_tablet_manager.cc:1397] T b788452af6154bfb985175c9ab2de563 P 974ee5dfbcb449f8849a6e84a6fff395: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260807 22:57:08.160899 5409 tablet_bootstrap.cc:654] T b788452af6154bfb985175c9ab2de563 P c9f865cfe1654d8f88a210773ef5d9bd: Neither blocks nor log segments found. Creating new log.
I20260807 22:57:08.161062 5411 tablet_bootstrap.cc:492] T b788452af6154bfb985175c9ab2de563 P 00fdb2f332f3437ea55e22ad5bf8ad14: Bootstrap starting.
I20260807 22:57:08.161852 5408 raft_consensus.cc:348] T b788452af6154bfb985175c9ab2de563 P 974ee5dfbcb449f8849a6e84a6fff395 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f865cfe1654d8f88a210773ef5d9bd" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 36449 } } peers { permanent_uuid: "974ee5dfbcb449f8849a6e84a6fff395" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 37625 } } peers { permanent_uuid: "00fdb2f332f3437ea55e22ad5bf8ad14" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 38459 } }
I20260807 22:57:08.162216 5408 raft_consensus.cc:374] T b788452af6154bfb985175c9ab2de563 P 974ee5dfbcb449f8849a6e84a6fff395 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:57:08.162328 5408 raft_consensus.cc:729] T b788452af6154bfb985175c9ab2de563 P 974ee5dfbcb449f8849a6e84a6fff395 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 974ee5dfbcb449f8849a6e84a6fff395, State: Initialized, Role: FOLLOWER
I20260807 22:57:08.162705 5408 consensus_queue.cc:260] T b788452af6154bfb985175c9ab2de563 P 974ee5dfbcb449f8849a6e84a6fff395 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f865cfe1654d8f88a210773ef5d9bd" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 36449 } } peers { permanent_uuid: "974ee5dfbcb449f8849a6e84a6fff395" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 37625 } } peers { permanent_uuid: "00fdb2f332f3437ea55e22ad5bf8ad14" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 38459 } }
I20260807 22:57:08.164011 5411 tablet_bootstrap.cc:654] T b788452af6154bfb985175c9ab2de563 P 00fdb2f332f3437ea55e22ad5bf8ad14: Neither blocks nor log segments found. Creating new log.
I20260807 22:57:08.165405 5408 ts_tablet_manager.cc:1428] T b788452af6154bfb985175c9ab2de563 P 974ee5dfbcb449f8849a6e84a6fff395: Time spent starting tablet: real 0.004s user 0.000s sys 0.005s
I20260807 22:57:08.165710 5409 tablet_bootstrap.cc:492] T b788452af6154bfb985175c9ab2de563 P c9f865cfe1654d8f88a210773ef5d9bd: No bootstrap required, opened a new log
I20260807 22:57:08.165891 5409 ts_tablet_manager.cc:1397] T b788452af6154bfb985175c9ab2de563 P c9f865cfe1654d8f88a210773ef5d9bd: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260807 22:57:08.167032 5409 raft_consensus.cc:348] T b788452af6154bfb985175c9ab2de563 P c9f865cfe1654d8f88a210773ef5d9bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f865cfe1654d8f88a210773ef5d9bd" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 36449 } } peers { permanent_uuid: "974ee5dfbcb449f8849a6e84a6fff395" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 37625 } } peers { permanent_uuid: "00fdb2f332f3437ea55e22ad5bf8ad14" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 38459 } }
I20260807 22:57:08.167398 5409 raft_consensus.cc:374] T b788452af6154bfb985175c9ab2de563 P c9f865cfe1654d8f88a210773ef5d9bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:57:08.167523 5409 raft_consensus.cc:729] T b788452af6154bfb985175c9ab2de563 P c9f865cfe1654d8f88a210773ef5d9bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9f865cfe1654d8f88a210773ef5d9bd, State: Initialized, Role: FOLLOWER
I20260807 22:57:08.167886 5409 consensus_queue.cc:260] T b788452af6154bfb985175c9ab2de563 P c9f865cfe1654d8f88a210773ef5d9bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f865cfe1654d8f88a210773ef5d9bd" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 36449 } } peers { permanent_uuid: "974ee5dfbcb449f8849a6e84a6fff395" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 37625 } } peers { permanent_uuid: "00fdb2f332f3437ea55e22ad5bf8ad14" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 38459 } }
I20260807 22:57:08.170066 5265 heartbeater.cc:503] Master 127.25.94.60:40149 was elected leader, sending a full tablet report...
I20260807 22:57:08.173754 5409 ts_tablet_manager.cc:1428] T b788452af6154bfb985175c9ab2de563 P c9f865cfe1654d8f88a210773ef5d9bd: Time spent starting tablet: real 0.008s user 0.004s sys 0.004s
I20260807 22:57:08.175239 5425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:08.174112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:32777 (local address 127.25.94.2:41701)
0807 22:57:08.174774 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:08.174943 (+ 169us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.62:32777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":533,"threads_started":1}
I20260807 22:57:08.177249 5425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:08.176646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:32777 (local address 127.25.94.2:52935)
0807 22:57:08.176907 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:08.177020 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.62:32777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128}
I20260807 22:57:08.179489 5419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:08.166098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:32777 (local address 127.25.94.3:57179)
0807 22:57:08.178269 (+ 12171us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:08.179032 (+ 763us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.62:32777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":11987,"thread_start_us":11774,"threads_started":1}
I20260807 22:57:08.180897 5419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:08.180295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:32777 (local address 127.25.94.3:46707)
0807 22:57:08.180596 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:08.180703 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.62:32777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147}
I20260807 22:57:08.181603 5411 tablet_bootstrap.cc:492] T b788452af6154bfb985175c9ab2de563 P 00fdb2f332f3437ea55e22ad5bf8ad14: No bootstrap required, opened a new log
I20260807 22:57:08.181816 5411 ts_tablet_manager.cc:1397] T b788452af6154bfb985175c9ab2de563 P 00fdb2f332f3437ea55e22ad5bf8ad14: Time spent bootstrapping tablet: real 0.021s user 0.009s sys 0.009s
I20260807 22:57:08.182647 5411 raft_consensus.cc:348] T b788452af6154bfb985175c9ab2de563 P 00fdb2f332f3437ea55e22ad5bf8ad14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f865cfe1654d8f88a210773ef5d9bd" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 36449 } } peers { permanent_uuid: "974ee5dfbcb449f8849a6e84a6fff395" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 37625 } } peers { permanent_uuid: "00fdb2f332f3437ea55e22ad5bf8ad14" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 38459 } }
I20260807 22:57:08.182946 5411 raft_consensus.cc:374] T b788452af6154bfb985175c9ab2de563 P 00fdb2f332f3437ea55e22ad5bf8ad14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:57:08.183053 5411 raft_consensus.cc:729] T b788452af6154bfb985175c9ab2de563 P 00fdb2f332f3437ea55e22ad5bf8ad14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00fdb2f332f3437ea55e22ad5bf8ad14, State: Initialized, Role: FOLLOWER
I20260807 22:57:08.183338 5411 consensus_queue.cc:260] T b788452af6154bfb985175c9ab2de563 P 00fdb2f332f3437ea55e22ad5bf8ad14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f865cfe1654d8f88a210773ef5d9bd" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 36449 } } peers { permanent_uuid: "974ee5dfbcb449f8849a6e84a6fff395" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 37625 } } peers { permanent_uuid: "00fdb2f332f3437ea55e22ad5bf8ad14" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 38459 } }
I20260807 22:57:08.184227 5411 ts_tablet_manager.cc:1428] T b788452af6154bfb985175c9ab2de563 P 00fdb2f332f3437ea55e22ad5bf8ad14: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260807 22:57:08.185877 5429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:08.184844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:32777 (local address 127.25.94.1:48291)
0807 22:57:08.185503 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:08.185628 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.62:32777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":382,"threads_started":1}
I20260807 22:57:08.186864 4780 heartbeater.cc:503] Master 127.25.94.60:40149 was elected leader, sending a full tablet report...
I20260807 22:57:08.187201 5429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:08.186753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:32777 (local address 127.25.94.1:42771)
0807 22:57:08.186960 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:08.187052 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.62:32777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260807 22:57:08.281468 5274 tablet.cc:2369] T b788452af6154bfb985175c9ab2de563 P 974ee5dfbcb449f8849a6e84a6fff395: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 22:57:08.336305 5414 raft_consensus.cc:482] T b788452af6154bfb985175c9ab2de563 P 974ee5dfbcb449f8849a6e84a6fff395 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 22:57:08.336584 5414 raft_consensus.cc:504] T b788452af6154bfb985175c9ab2de563 P 974ee5dfbcb449f8849a6e84a6fff395 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f865cfe1654d8f88a210773ef5d9bd" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 36449 } } peers { permanent_uuid: "974ee5dfbcb449f8849a6e84a6fff395" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 37625 } } peers { permanent_uuid: "00fdb2f332f3437ea55e22ad5bf8ad14" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 38459 } }
I20260807 22:57:08.338550 5414 leader_election.cc:288] T b788452af6154bfb985175c9ab2de563 P 974ee5dfbcb449f8849a6e84a6fff395 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9f865cfe1654d8f88a210773ef5d9bd (127.25.94.2:36449), 00fdb2f332f3437ea55e22ad5bf8ad14 (127.25.94.1:38459)
I20260807 22:57:08.345844 5419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:08.338877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.2:36449 (local address 127.25.94.3:37045)
0807 22:57:08.339075 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:08.339095 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 22:57:08.339259 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:57:08.339678 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:57:08.339684 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:57:08.339732 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:57:08.340097 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:08.340109 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:08.341619 (+ 1510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:08.341625 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:08.343505 (+ 1880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:08.343519 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:08.345019 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:08.345027 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:08.345251 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:08.345289 (+ 38us) client_negotiation.cc:769] Sending connection context
0807 22:57:08.345383 (+ 94us) client_negotiation.cc:240] Negotiation successful
0807 22:57:08.345527 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260807 22:57:08.345873 5404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:08.339146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.3:37045 (local address 127.25.94.2:36449)
0807 22:57:08.339393 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:08.339406 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:08.339424 (+ 18us) server_negotiation.cc:407] Connection header received
0807 22:57:08.339481 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:08.339484 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:08.339541 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:08.339618 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:57:08.340321 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:08.341481 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:08.343670 (+ 2189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:08.344830 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:08.344966 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:08.345073 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0807 22:57:08.345435 (+ 362us) server_negotiation.cc:299] Negotiation successful
0807 22:57:08.345570 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260807 22:57:08.346855 4954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b788452af6154bfb985175c9ab2de563" candidate_uuid: "974ee5dfbcb449f8849a6e84a6fff395" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9f865cfe1654d8f88a210773ef5d9bd" is_pre_election: true
I20260807 22:57:08.347385 5430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:08.338877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.1:38459 (local address 127.25.94.3:54275)
0807 22:57:08.340441 (+ 1564us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:08.340456 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 22:57:08.340598 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:57:08.340944 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:57:08.340949 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:57:08.340971 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:57:08.341301 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:08.341310 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:08.342688 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:08.342694 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:08.344722 (+ 2028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:08.344738 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:08.346580 (+ 1842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:08.346586 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:08.346791 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:08.346831 (+ 40us) client_negotiation.cc:769] Sending connection context
0807 22:57:08.346940 (+ 109us) client_negotiation.cc:240] Negotiation successful
0807 22:57:08.347051 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1459,"mutex_wait_us":839,"thread_start_us":398,"threads_started":1}
I20260807 22:57:08.347395 5407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:08.339206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.3:54275 (local address 127.25.94.1:38459)
0807 22:57:08.339432 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:08.339437 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:08.340549 (+ 1112us) server_negotiation.cc:407] Connection header received
0807 22:57:08.340743 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:08.340749 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:08.340828 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:08.340930 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:57:08.341444 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:08.342531 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:08.344892 (+ 2361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:08.346377 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:08.346507 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:08.346633 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0807 22:57:08.346991 (+ 358us) server_negotiation.cc:299] Negotiation successful
0807 22:57:08.347140 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260807 22:57:08.347373 4954 raft_consensus.cc:2454] T b788452af6154bfb985175c9ab2de563 P c9f865cfe1654d8f88a210773ef5d9bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974ee5dfbcb449f8849a6e84a6fff395 in term 0.
I20260807 22:57:08.347980 5084 leader_election.cc:302] T b788452af6154bfb985175c9ab2de563 P 974ee5dfbcb449f8849a6e84a6fff395 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 974ee5dfbcb449f8849a6e84a6fff395, c9f865cfe1654d8f88a210773ef5d9bd; no voters:
I20260807 22:57:08.348172 4702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b788452af6154bfb985175c9ab2de563" candidate_uuid: "974ee5dfbcb449f8849a6e84a6fff395" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fdb2f332f3437ea55e22ad5bf8ad14" is_pre_election: true
I20260807 22:57:08.348395 5414 raft_consensus.cc:2781] T b788452af6154bfb985175c9ab2de563 P 974ee5dfbcb449f8849a6e84a6fff395 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 22:57:08.348529 5414 raft_consensus.cc:482] T b788452af6154bfb985175c9ab2de563 P 974ee5dfbcb449f8849a6e84a6fff395 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 22:57:08.348500 4702 raft_consensus.cc:2454] T b788452af6154bfb985175c9ab2de563 P 00fdb2f332f3437ea55e22ad5bf8ad14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974ee5dfbcb449f8849a6e84a6fff395 in term 0.
I20260807 22:57:08.348618 5414 raft_consensus.cc:3037] T b788452af6154bfb985175c9ab2de563 P 974ee5dfbcb449f8849a6e84a6fff395 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:57:08.350884 5414 raft_consensus.cc:504] T b788452af6154bfb985175c9ab2de563 P 974ee5dfbcb449f8849a6e84a6fff395 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f865cfe1654d8f88a210773ef5d9bd" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 36449 } } peers { permanent_uuid: "974ee5dfbcb449f8849a6e84a6fff395" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 37625 } } peers { permanent_uuid: "00fdb2f332f3437ea55e22ad5bf8ad14" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 38459 } }
I20260807 22:57:08.351611 5414 leader_election.cc:288] T b788452af6154bfb985175c9ab2de563 P 974ee5dfbcb449f8849a6e84a6fff395 [CANDIDATE]: Term 1 election: Requested vote from peers c9f865cfe1654d8f88a210773ef5d9bd (127.25.94.2:36449), 00fdb2f332f3437ea55e22ad5bf8ad14 (127.25.94.1:38459)
I20260807 22:57:08.352072 4954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b788452af6154bfb985175c9ab2de563" candidate_uuid: "974ee5dfbcb449f8849a6e84a6fff395" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9f865cfe1654d8f88a210773ef5d9bd"
I20260807 22:57:08.352195 4702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b788452af6154bfb985175c9ab2de563" candidate_uuid: "974ee5dfbcb449f8849a6e84a6fff395" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fdb2f332f3437ea55e22ad5bf8ad14"
I20260807 22:57:08.352356 4954 raft_consensus.cc:3037] T b788452af6154bfb985175c9ab2de563 P c9f865cfe1654d8f88a210773ef5d9bd [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:57:08.352383 4702 raft_consensus.cc:3037] T b788452af6154bfb985175c9ab2de563 P 00fdb2f332f3437ea55e22ad5bf8ad14 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:57:08.355499 4702 raft_consensus.cc:2454] T b788452af6154bfb985175c9ab2de563 P 00fdb2f332f3437ea55e22ad5bf8ad14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974ee5dfbcb449f8849a6e84a6fff395 in term 1.
I20260807 22:57:08.355499 4954 raft_consensus.cc:2454] T b788452af6154bfb985175c9ab2de563 P c9f865cfe1654d8f88a210773ef5d9bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974ee5dfbcb449f8849a6e84a6fff395 in term 1.
I20260807 22:57:08.355976 5084 leader_election.cc:302] T b788452af6154bfb985175c9ab2de563 P 974ee5dfbcb449f8849a6e84a6fff395 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 974ee5dfbcb449f8849a6e84a6fff395, c9f865cfe1654d8f88a210773ef5d9bd; no voters:
I20260807 22:57:08.356308 5414 raft_consensus.cc:2781] T b788452af6154bfb985175c9ab2de563 P 974ee5dfbcb449f8849a6e84a6fff395 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:57:08.356809 5414 raft_consensus.cc:686] T b788452af6154bfb985175c9ab2de563 P 974ee5dfbcb449f8849a6e84a6fff395 [term 1 LEADER]: Becoming Leader. State: Replica: 974ee5dfbcb449f8849a6e84a6fff395, State: Running, Role: LEADER
I20260807 22:57:08.357326 5414 consensus_queue.cc:237] T b788452af6154bfb985175c9ab2de563 P 974ee5dfbcb449f8849a6e84a6fff395 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f865cfe1654d8f88a210773ef5d9bd" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 36449 } } peers { permanent_uuid: "974ee5dfbcb449f8849a6e84a6fff395" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 37625 } } peers { permanent_uuid: "00fdb2f332f3437ea55e22ad5bf8ad14" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 38459 } }
I20260807 22:57:08.358110 5430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:08.357715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:32777 (local address 127.25.94.3:48343)
0807 22:57:08.357867 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:08.357953 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.62:32777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260807 22:57:08.361111 4468 catalog_manager.cc:5495] T b788452af6154bfb985175c9ab2de563 P 974ee5dfbcb449f8849a6e84a6fff395 reported cstate change: term changed from 0 to 1, leader changed from <none> to 974ee5dfbcb449f8849a6e84a6fff395 (127.25.94.3). New cstate: current_term: 1 leader_uuid: "974ee5dfbcb449f8849a6e84a6fff395" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9f865cfe1654d8f88a210773ef5d9bd" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 36449 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "974ee5dfbcb449f8849a6e84a6fff395" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 37625 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00fdb2f332f3437ea55e22ad5bf8ad14" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 38459 } health_report { overall_health: UNKNOWN } } }
I20260807 22:57:08.395744 5405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:08.390466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48496 (local address 127.25.94.3:37625)
0807 22:57:08.390732 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:08.390740 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:08.390763 (+ 23us) server_negotiation.cc:407] Connection header received
0807 22:57:08.390875 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:08.390880 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:08.390958 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:08.391124 (+ 166us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 22:57:08.391504 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:08.392229 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:08.393577 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:08.393995 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:08.394137 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:08.394562 (+ 425us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 22:57:08.395126 (+ 564us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 22:57:08.395285 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0807 22:57:08.395384 (+ 99us) server_negotiation.cc:299] Negotiation successful
0807 22:57:08.395468 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260807 22:57:08.400178 4702 raft_consensus.cc:1261] T b788452af6154bfb985175c9ab2de563 P 00fdb2f332f3437ea55e22ad5bf8ad14 [term 1 FOLLOWER]: Refusing update from remote peer 974ee5dfbcb449f8849a6e84a6fff395: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 22:57:08.400482 4954 raft_consensus.cc:1261] T b788452af6154bfb985175c9ab2de563 P c9f865cfe1654d8f88a210773ef5d9bd [term 1 FOLLOWER]: Refusing update from remote peer 974ee5dfbcb449f8849a6e84a6fff395: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 22:57:08.400962 5432 consensus_queue.cc:1035] T b788452af6154bfb985175c9ab2de563 P 974ee5dfbcb449f8849a6e84a6fff395 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fdb2f332f3437ea55e22ad5bf8ad14" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 38459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 22:57:08.401360 5414 consensus_queue.cc:1035] T b788452af6154bfb985175c9ab2de563 P 974ee5dfbcb449f8849a6e84a6fff395 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9f865cfe1654d8f88a210773ef5d9bd" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 36449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 22:57:08.404001 5031 heartbeater.cc:503] Master 127.25.94.60:40149 was elected leader, sending a full tablet report...
I20260807 22:57:08.405435 5429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:08.403658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:32777 (local address 127.25.94.1:45691)
0807 22:57:08.404999 (+ 1341us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:08.405160 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.62:32777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1146}
I20260807 22:57:08.406024 5425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:08.405563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:32777 (local address 127.25.94.2:48037)
0807 22:57:08.405727 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:08.405818 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.62:32777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260807 22:57:08.410696 5429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:08.410242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:32777 (local address 127.25.94.1:39501)
0807 22:57:08.410460 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:08.410550 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.62:32777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
I20260807 22:57:08.413249 5425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:08.412832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:32777 (local address 127.25.94.2:37743)
0807 22:57:08.412997 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:08.413070 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.62:32777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260807 22:57:08.423485 5430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:08.422839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:32777 (local address 127.25.94.3:45553)
0807 22:57:08.423107 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:08.423225 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.62:32777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
I20260807 22:57:08.430351 4468 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52410:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 22:57:08.430678 4468 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52410:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 22:57:08.442236 4669 tablet_service.cc:1514] Processing DeleteTablet for tablet b788452af6154bfb985175c9ab2de563 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:57:08 UTC) from {username='oryx', principal='oryx/127.25.94.60@KRBTEST.COM'} at 127.0.0.1:45536
I20260807 22:57:08.442919 4468 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6b8de2d663914806a8d5fc4fb0eda2b9: Sending DeleteTablet for 3 replicas of tablet b788452af6154bfb985175c9ab2de563
I20260807 22:57:08.442996 4924 tablet_service.cc:1514] Processing DeleteTablet for tablet b788452af6154bfb985175c9ab2de563 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:57:08 UTC) from {username='oryx', principal='oryx/127.25.94.60@KRBTEST.COM'} at 127.0.0.1:35068
I20260807 22:57:08.444355 4667 tablet_service.cc:1514] Processing DeleteTablet for tablet b788452af6154bfb985175c9ab2de563 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:57:08 UTC) from {username='oryx', principal='oryx/127.25.94.60@KRBTEST.COM'} at 127.0.0.1:45536
I20260807 22:57:08.445652 5153 tablet_service.cc:1514] Processing DeleteTablet for tablet b788452af6154bfb985175c9ab2de563 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:57:08 UTC) from {username='oryx', principal='oryx/127.25.94.60@KRBTEST.COM'} at 127.0.0.1:48488
I20260807 22:57:08.446496 4923 tablet_service.cc:1514] Processing DeleteTablet for tablet b788452af6154bfb985175c9ab2de563 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:57:08 UTC) from {username='oryx', principal='oryx/127.25.94.60@KRBTEST.COM'} at 127.0.0.1:35068
I20260807 22:57:08.447947 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 4424
I20260807 22:57:08.455992 5458 tablet_replica.cc:330] T b788452af6154bfb985175c9ab2de563 P 974ee5dfbcb449f8849a6e84a6fff395: stopping tablet replica
I20260807 22:57:08.456456 5458 raft_consensus.cc:2229] T b788452af6154bfb985175c9ab2de563 P 974ee5dfbcb449f8849a6e84a6fff395 [term 1 LEADER]: Raft consensus shutting down.
I20260807 22:57:08.457425 5458 raft_consensus.cc:2258] T b788452af6154bfb985175c9ab2de563 P 974ee5dfbcb449f8849a6e84a6fff395 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 22:57:08.458889 5458 ts_tablet_manager.cc:1905] T b788452af6154bfb985175c9ab2de563 P 974ee5dfbcb449f8849a6e84a6fff395: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 22:57:08.461668 5456 tablet_replica.cc:330] T b788452af6154bfb985175c9ab2de563 P 00fdb2f332f3437ea55e22ad5bf8ad14: stopping tablet replica
I20260807 22:57:08.461890 5456 raft_consensus.cc:2229] T b788452af6154bfb985175c9ab2de563 P 00fdb2f332f3437ea55e22ad5bf8ad14 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 22:57:08.462324 5456 raft_consensus.cc:2258] T b788452af6154bfb985175c9ab2de563 P 00fdb2f332f3437ea55e22ad5bf8ad14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 22:57:08.464468 5456 ts_tablet_manager.cc:1905] T b788452af6154bfb985175c9ab2de563 P 00fdb2f332f3437ea55e22ad5bf8ad14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 22:57:08.465092 5457 tablet_replica.cc:330] T b788452af6154bfb985175c9ab2de563 P c9f865cfe1654d8f88a210773ef5d9bd: stopping tablet replica
I20260807 22:57:08.465335 5457 raft_consensus.cc:2229] T b788452af6154bfb985175c9ab2de563 P c9f865cfe1654d8f88a210773ef5d9bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 22:57:08.465698 5457 raft_consensus.cc:2258] T b788452af6154bfb985175c9ab2de563 P c9f865cfe1654d8f88a210773ef5d9bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 22:57:08.468358 5458 ts_tablet_manager.cc:1918] T b788452af6154bfb985175c9ab2de563 P 974ee5dfbcb449f8849a6e84a6fff395: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 22:57:08.468533 5458 log.cc:1196] T b788452af6154bfb985175c9ab2de563 P 974ee5dfbcb449f8849a6e84a6fff395: Deleting WAL directory at /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/ts-2/wal/wals/b788452af6154bfb985175c9ab2de563
I20260807 22:57:08.469089 5458 ts_tablet_manager.cc:1939] T b788452af6154bfb985175c9ab2de563 P 974ee5dfbcb449f8849a6e84a6fff395: Deleting consensus metadata
I20260807 22:57:08.470536 4449 catalog_manager.cc:4841] TS 974ee5dfbcb449f8849a6e84a6fff395 (127.25.94.3:37625): tablet b788452af6154bfb985175c9ab2de563 (table test-table [id=3a526c6fe3604845b77f996351bc3b89]) successfully deleted
I20260807 22:57:08.470603 5456 ts_tablet_manager.cc:1918] T b788452af6154bfb985175c9ab2de563 P 00fdb2f332f3437ea55e22ad5bf8ad14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 22:57:08.470742 5456 log.cc:1196] T b788452af6154bfb985175c9ab2de563 P 00fdb2f332f3437ea55e22ad5bf8ad14: Deleting WAL directory at /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/ts-0/wal/wals/b788452af6154bfb985175c9ab2de563
I20260807 22:57:08.471256 5456 ts_tablet_manager.cc:1939] T b788452af6154bfb985175c9ab2de563 P 00fdb2f332f3437ea55e22ad5bf8ad14: Deleting consensus metadata
I20260807 22:57:08.471488 5457 ts_tablet_manager.cc:1905] T b788452af6154bfb985175c9ab2de563 P c9f865cfe1654d8f88a210773ef5d9bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 22:57:08.472560 4452 catalog_manager.cc:4841] TS 00fdb2f332f3437ea55e22ad5bf8ad14 (127.25.94.1:38459): tablet b788452af6154bfb985175c9ab2de563 (table test-table [id=3a526c6fe3604845b77f996351bc3b89]) successfully deleted
W20260807 22:57:08.473429 4452 catalog_manager.cc:4808] TS 00fdb2f332f3437ea55e22ad5bf8ad14 (127.25.94.1:38459): delete failed for tablet b788452af6154bfb985175c9ab2de563 because the tablet was not found. No further retry: Not found: Tablet not found: b788452af6154bfb985175c9ab2de563
I20260807 22:57:08.479323 5457 ts_tablet_manager.cc:1918] T b788452af6154bfb985175c9ab2de563 P c9f865cfe1654d8f88a210773ef5d9bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 22:57:08.479516 5457 log.cc:1196] T b788452af6154bfb985175c9ab2de563 P c9f865cfe1654d8f88a210773ef5d9bd: Deleting WAL directory at /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786143361779964-25976-0/minicluster-data/ts-1/wal/wals/b788452af6154bfb985175c9ab2de563
I20260807 22:57:08.480032 5457 ts_tablet_manager.cc:1939] T b788452af6154bfb985175c9ab2de563 P c9f865cfe1654d8f88a210773ef5d9bd: Deleting consensus metadata
I20260807 22:57:08.481462 4449 catalog_manager.cc:4841] TS c9f865cfe1654d8f88a210773ef5d9bd (127.25.94.2:36449): tablet b788452af6154bfb985175c9ab2de563 (table test-table [id=3a526c6fe3604845b77f996351bc3b89]) successfully deleted
W20260807 22:57:08.482424 4449 catalog_manager.cc:4808] TS c9f865cfe1654d8f88a210773ef5d9bd (127.25.94.2:36449): delete failed for tablet b788452af6154bfb985175c9ab2de563 because the tablet was not found. No further retry: Not found: Tablet not found: b788452af6154bfb985175c9ab2de563
I20260807 22:57:08.491973 5407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:08.449328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45548 (local address 127.25.94.1:38459)
0807 22:57:08.449520 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:08.449526 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:08.475130 (+ 25604us) server_negotiation.cc:407] Connection header received
0807 22:57:08.475381 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:08.475407 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:08.475490 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:08.475587 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:08.477921 (+ 2334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:08.479059 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:08.480740 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:08.481311 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:08.481441 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:08.485472 (+ 4031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:57:08.485504 (+ 32us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:57:08.485521 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:57:08.485580 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:57:08.488784 (+ 3204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:08.489835 (+ 1051us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:08.489843 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:08.489845 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:08.489998 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:08.490422 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:08.490425 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:08.490427 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:08.490831 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:57:08.491061 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0807 22:57:08.491450 (+ 389us) server_negotiation.cc:299] Negotiation successful
0807 22:57:08.491631 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260807 22:57:08.492787 4772 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 22:57:08.591156 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 4424
I20260807 22:57:08.623958 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 4815
I20260807 22:57:08.640403 5404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:08.624790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35074 (local address 127.25.94.2:36449)
0807 22:57:08.624968 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:08.624976 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:08.625058 (+ 82us) server_negotiation.cc:407] Connection header received
0807 22:57:08.625305 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:08.625310 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:08.625393 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:08.625503 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:08.626845 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:08.627583 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:08.628838 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:08.629432 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:08.629585 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:08.633001 (+ 3416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:57:08.633018 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:57:08.633035 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:57:08.633085 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:57:08.636543 (+ 3458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:08.637927 (+ 1384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:08.637940 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:08.637943 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:08.638110 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:08.638590 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:08.638596 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:08.638599 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:08.639021 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:57:08.639261 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0807 22:57:08.639716 (+ 455us) server_negotiation.cc:299] Negotiation successful
0807 22:57:08.639979 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260807 22:57:08.641129 5023 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 22:57:08.657042 5406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:08.656537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:32777 (local address 127.0.0.1:41848)
0807 22:57:08.656738 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:08.656852 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.62:32777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260807 22:57:08.674017 5406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:08.673430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:32777 (local address 127.0.0.1:41854)
0807 22:57:08.673680 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:08.673832 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.62:32777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260807 22:57:08.775971 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 4815
I20260807 22:57:08.817955 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 5055
I20260807 22:57:08.835352 5405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:08.818762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48504 (local address 127.25.94.3:37625)
0807 22:57:08.818970 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:08.818976 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:08.819000 (+ 24us) server_negotiation.cc:407] Connection header received
0807 22:57:08.819184 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:08.819190 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:08.819287 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:08.819401 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:08.820865 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:08.822283 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:08.823598 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:08.824362 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:08.824508 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:08.827256 (+ 2748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:57:08.827277 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:57:08.827306 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:57:08.827350 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:57:08.831972 (+ 4622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:08.832766 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:08.832776 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:08.832779 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:08.832980 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:08.833526 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:08.833538 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:08.833541 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:08.833954 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:57:08.834373 (+ 419us) server_negotiation.cc:1035] Waiting for connection context
0807 22:57:08.834729 (+ 356us) server_negotiation.cc:299] Negotiation successful
0807 22:57:08.834980 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260807 22:57:08.836126 5259 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 22:57:08.968911 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 5055
I20260807 22:57:08.994561 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 3930
I20260807 22:57:08.995755 4214 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 22:57:09.087771 5406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:09.087307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:32777 (local address 127.0.0.1:41860)
0807 22:57:09.087509 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:09.087625 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.62:32777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260807 22:57:09.088768 5406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:09.088441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:32777 (local address 127.0.0.1:41876)
0807 22:57:09.088583 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:09.088657 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.62:32777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260807 22:57:09.116236 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 3930
I20260807 22:57:09.148137 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 4102
I20260807 22:57:09.149361 4526 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 22:57:09.273976 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 4102
I20260807 22:57:09.274020 5406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:09.273481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.61:45205 (local address 127.0.0.1:57890)
0807 22:57:09.273720 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:09.273852 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.61:45205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
2026-08-07T22:57:09Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (10933 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786143361779964-25976-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 22:57:09 dist-test-slave-z8x9 krb5kdc[5473](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 22:57:09 dist-test-slave-z8x9 krb5kdc[5473](info): set up 2 sockets
Aug 07 22:57:09 dist-test-slave-z8x9 krb5kdc[5473](info): commencing operation
krb5kdc: starting...
W20260807 22:57:10.650660 25976 mini_kdc.cc:121] Time spent starting KDC: real 1.321s user 0.003s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786143361779964-25976-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786143361779964-25976-0/krb5kdc/test-user.keytab.
Aug 07 22:57:10 dist-test-slave-z8x9 krb5kdc[5473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143430, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T22:57:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T22:57:10Z Disabled control of system clock
WARNING: no policy specified for oryx/127.25.94.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.25.94.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.25.94.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786143361779964-25976-0/krb5kdc/oryx_127.25.94.62.keytab.
Entry for principal oryx/127.25.94.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786143361779964-25976-0/krb5kdc/oryx_127.25.94.62.keytab.
I20260807 22:57:10.916469 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786143361779964-25976-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786143361779964-25976-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786143361779964-25976-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.62:34919
--webserver_interface=127.25.94.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786143361779964-25976-0/krb5kdc/oryx_127.25.94.62.keytab
--principal=oryx/127.25.94.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.20:39623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.94.62:34919
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786143361779964-25976-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786143361779964-25976-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786143361779964-25976-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:57:11.265970 5508 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:57:11.266289 5508 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:57:11.266460 5508 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:57:11.273856 5508 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 22:57:11.273985 5508 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260807 22:57:11.274075 5508 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:57:11.274175 5508 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:57:11.274266 5508 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 22:57:11.274389 5508 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 22:57:11.284788 5508 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:39623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786143361779964-25976-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786143361779964-25976-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.94.62:34919
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786143361779964-25976-0/krb5kdc/oryx_127.25.94.62.keytab
--principal=oryx/127.25.94.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.62:34919
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.94.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786143361779964-25976-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 22:57:11.286232 5508 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:57:11.287655 5508 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:57:11.300103 5525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:57:11.300063 5521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:57:11.300153 5518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:57:11.300609 5508 server_base.cc:1034] running on GCE node
I20260807 22:57:11.301721 5508 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:57:11.303083 5508 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:57:11.304319 5508 hybrid_clock.cc:648] HybridClock initialized: now 1786143431304252 us; error 34 us; skew 500 ppm
I20260807 22:57:11.304636 5508 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 22:57:11 dist-test-slave-z8x9 krb5kdc[5473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143431, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.94.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:57:11.312487 5508 init.cc:377] Logged in from keytab as oryx/127.25.94.62@KRBTEST.COM (short username oryx)
I20260807 22:57:11.314388 5508 webserver.cc:460] Webserver started at http://127.25.94.62:45567/ using document root <none> and password file <none>
I20260807 22:57:11.315333 5508 fs_manager.cc:359] Metadata directory not provided
I20260807 22:57:11.315445 5508 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:57:11.315742 5508 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:57:11.318966 5508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786143361779964-25976-0/minicluster-data/master-0/data/instance:
uuid: "1f81ab1e409a4b718bf275a1e92f4448"
format_stamp: "Formatted at 2026-08-07 22:57:11 on dist-test-slave-z8x9"
server_key: "89962b8c8d2132c051b40a5ea1df3cb0"
server_key_iv: "44fe645d9503aa784d7f3577bfe9d551"
server_key_version: "encryptionkey@0"
I20260807 22:57:11.319734 5508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786143361779964-25976-0/minicluster-data/master-0/wal/instance:
uuid: "1f81ab1e409a4b718bf275a1e92f4448"
format_stamp: "Formatted at 2026-08-07 22:57:11 on dist-test-slave-z8x9"
server_key: "89962b8c8d2132c051b40a5ea1df3cb0"
server_key_iv: "44fe645d9503aa784d7f3577bfe9d551"
server_key_version: "encryptionkey@0"
I20260807 22:57:11.325098 5508 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260807 22:57:11.328751 5534 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:57:11.330276 5508 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 22:57:11.330441 5508 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786143361779964-25976-0/minicluster-data/master-0/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786143361779964-25976-0/minicluster-data/master-0/wal
uuid: "1f81ab1e409a4b718bf275a1e92f4448"
format_stamp: "Formatted at 2026-08-07 22:57:11 on dist-test-slave-z8x9"
server_key: "89962b8c8d2132c051b40a5ea1df3cb0"
server_key_iv: "44fe645d9503aa784d7f3577bfe9d551"
server_key_version: "encryptionkey@0"
I20260807 22:57:11.330633 5508 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786143361779964-25976-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786143361779964-25976-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786143361779964-25976-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:57:11.342417 5508 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:57:11.346590 5508 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:57:11.346833 5508 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:57:11.369676 5508 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.62:34919
I20260807 22:57:11.369696 5630 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.62:34919 every 8 connection(s)
I20260807 22:57:11.371145 5508 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
I20260807 22:57:11.372282 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 5508
I20260807 22:57:11.372522 25976 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786143361779964-25976-0/minicluster-data/master-0/wal/instance
I20260807 22:57:11.373394 25976 external_mini_cluster.cc:1428] Setting key a3bc01a6a70b18ea7b9e20748bf5169a
I20260807 22:57:11.376164 5631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:57:11.387825 5631 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f81ab1e409a4b718bf275a1e92f4448: Bootstrap starting.
Aug 07 22:57:11 dist-test-slave-z8x9 krb5kdc[5473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143430, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.25.94.62@KRBTEST.COM
I20260807 22:57:11.391631 5631 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1f81ab1e409a4b718bf275a1e92f4448: Neither blocks nor log segments found. Creating new log.
I20260807 22:57:11.392882 5631 log.cc:824] T 00000000000000000000000000000000 P 1f81ab1e409a4b718bf275a1e92f4448: Log is configured to *not* fsync() on all Append() calls
I20260807 22:57:11.396008 5631 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f81ab1e409a4b718bf275a1e92f4448: No bootstrap required, opened a new log
I20260807 22:57:11.397486 5641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:11.375605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53050 (local address 127.25.94.62:34919)
0807 22:57:11.376484 (+ 879us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:11.376497 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:11.376540 (+ 43us) server_negotiation.cc:407] Connection header received
0807 22:57:11.378273 (+ 1733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:11.378296 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:11.378709 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:11.379143 (+ 434us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:11.380763 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:11.382253 (+ 1490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:11.383551 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:11.384273 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:11.384511 (+ 238us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:11.389235 (+ 4724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:57:11.389250 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:57:11.389286 (+ 36us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:57:11.389351 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:57:11.393053 (+ 3702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:11.393890 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:11.393899 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:11.393920 (+ 21us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:11.394110 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:11.394536 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:11.394541 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:11.394545 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:11.395095 (+ 550us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:57:11.395416 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0807 22:57:11.395871 (+ 455us) server_negotiation.cc:299] Negotiation successful
0807 22:57:11.396355 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":645,"thread_start_us":366,"threads_started":1}
I20260807 22:57:11.403965 5631 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1f81ab1e409a4b718bf275a1e92f4448 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f81ab1e409a4b718bf275a1e92f4448" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 34919 } }
I20260807 22:57:11.404378 5631 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1f81ab1e409a4b718bf275a1e92f4448 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:57:11.404497 5631 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1f81ab1e409a4b718bf275a1e92f4448 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f81ab1e409a4b718bf275a1e92f4448, State: Initialized, Role: FOLLOWER
I20260807 22:57:11.405064 5631 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1f81ab1e409a4b718bf275a1e92f4448 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f81ab1e409a4b718bf275a1e92f4448" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 34919 } }
I20260807 22:57:11.405385 5631 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1f81ab1e409a4b718bf275a1e92f4448 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 22:57:11.405546 5631 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1f81ab1e409a4b718bf275a1e92f4448 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 22:57:11.405745 5631 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1f81ab1e409a4b718bf275a1e92f4448 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:57:11.408259 5631 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1f81ab1e409a4b718bf275a1e92f4448 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f81ab1e409a4b718bf275a1e92f4448" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 34919 } }
I20260807 22:57:11.408784 5631 leader_election.cc:302] T 00000000000000000000000000000000 P 1f81ab1e409a4b718bf275a1e92f4448 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1f81ab1e409a4b718bf275a1e92f4448; no voters:
I20260807 22:57:11.409456 5631 leader_election.cc:288] T 00000000000000000000000000000000 P 1f81ab1e409a4b718bf275a1e92f4448 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 22:57:11.409601 5643 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1f81ab1e409a4b718bf275a1e92f4448 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:57:11.410460 5643 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1f81ab1e409a4b718bf275a1e92f4448 [term 1 LEADER]: Becoming Leader. State: Replica: 1f81ab1e409a4b718bf275a1e92f4448, State: Running, Role: LEADER
I20260807 22:57:11.411062 5643 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1f81ab1e409a4b718bf275a1e92f4448 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f81ab1e409a4b718bf275a1e92f4448" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 34919 } }
I20260807 22:57:11.412027 5631 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1f81ab1e409a4b718bf275a1e92f4448 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 22:57:11.414601 5650 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f81ab1e409a4b718bf275a1e92f4448 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1f81ab1e409a4b718bf275a1e92f4448. Latest consensus state: current_term: 1 leader_uuid: "1f81ab1e409a4b718bf275a1e92f4448" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f81ab1e409a4b718bf275a1e92f4448" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 34919 } } }
I20260807 22:57:11.415131 5650 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f81ab1e409a4b718bf275a1e92f4448 [sys.catalog]: This master's current role is: LEADER
I20260807 22:57:11.415675 5646 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f81ab1e409a4b718bf275a1e92f4448 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1f81ab1e409a4b718bf275a1e92f4448" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f81ab1e409a4b718bf275a1e92f4448" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 34919 } } }
I20260807 22:57:11.415963 5646 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f81ab1e409a4b718bf275a1e92f4448 [sys.catalog]: This master's current role is: LEADER
I20260807 22:57:11.417555 5660 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 22:57:11.423502 5660 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 22:57:11.433007 5660 catalog_manager.cc:1347] Generated new cluster ID: b73925a1015f4d48a27842dd7de24543
I20260807 22:57:11.433147 5660 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 22:57:11.459381 5660 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 22:57:11.460794 5660 catalog_manager.cc:1504] Loading token signing keys...
I20260807 22:57:11.469081 5660 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1f81ab1e409a4b718bf275a1e92f4448: Generated new TSK 0
I20260807 22:57:11.470144 5660 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.25.94.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.25.94.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.25.94.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786143361779964-25976-0/krb5kdc/oryx_127.25.94.1.keytab.
Entry for principal oryx/127.25.94.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786143361779964-25976-0/krb5kdc/oryx_127.25.94.1.keytab.
I20260807 22:57:11.544885 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786143361779964-25976-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786143361779964-25976-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786143361779964-25976-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786143361779964-25976-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.1:0
--local_ip_for_outbound_sockets=127.25.94.1
--webserver_interface=127.25.94.1
--webserver_port=0
--tserver_master_addrs=127.25.94.62:34919
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786143361779964-25976-0/krb5kdc/oryx_127.25.94.1.keytab
--principal=oryx/127.25.94.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.20:39623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786143361779964-25976-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786143361779964-25976-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786143361779964-25976-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:57:11.886313 5678 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:57:11.886586 5678 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:57:11.886685 5678 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:57:11.893656 5678 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:57:11.893757 5678 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:57:11.893846 5678 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260807 22:57:11.893924 5678 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.1
I20260807 22:57:11.903893 5678 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:39623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786143361779964-25976-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786143361779964-25976-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786143361779964-25976-0/krb5kdc/oryx_127.25.94.1.keytab
--principal=oryx/127.25.94.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786143361779964-25976-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.94.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.25.94.62:34919
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.1
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786143361779964-25976-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 22:57:11.905104 5678 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:57:11.906450 5678 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:57:11.920394 5689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:57:11.920624 5686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:57:11.920589 5692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:57:11.921715 5678 server_base.cc:1034] running on GCE node
I20260807 22:57:11.922350 5678 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:57:11.923278 5678 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:57:11.924542 5678 hybrid_clock.cc:648] HybridClock initialized: now 1786143431924482 us; error 40 us; skew 500 ppm
I20260807 22:57:11.924863 5678 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 22:57:11 dist-test-slave-z8x9 krb5kdc[5473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143431, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.94.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:57:11.928131 5678 init.cc:377] Logged in from keytab as oryx/127.25.94.1@KRBTEST.COM (short username oryx)
I20260807 22:57:11.929772 5678 webserver.cc:460] Webserver started at http://127.25.94.1:44591/ using document root <none> and password file <none>
I20260807 22:57:11.930476 5678 fs_manager.cc:359] Metadata directory not provided
I20260807 22:57:11.930615 5678 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:57:11.930944 5678 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:57:11.933939 5678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786143361779964-25976-0/minicluster-data/ts-0/data/instance:
uuid: "6609f2fb140445708643ec0ce81dbdbf"
format_stamp: "Formatted at 2026-08-07 22:57:11 on dist-test-slave-z8x9"
server_key: "76de60eb0896075eb6c37b91ed50ef6f"
server_key_iv: "150296e9a7c9c964fb6f73a515f0c3eb"
server_key_version: "encryptionkey@0"
I20260807 22:57:11.934894 5678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786143361779964-25976-0/minicluster-data/ts-0/wal/instance:
uuid: "6609f2fb140445708643ec0ce81dbdbf"
format_stamp: "Formatted at 2026-08-07 22:57:11 on dist-test-slave-z8x9"
server_key: "76de60eb0896075eb6c37b91ed50ef6f"
server_key_iv: "150296e9a7c9c964fb6f73a515f0c3eb"
server_key_version: "encryptionkey@0"
I20260807 22:57:11.940196 5678 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260807 22:57:11.943840 5704 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:57:11.945158 5678 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 22:57:11.945405 5678 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786143361779964-25976-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786143361779964-25976-0/minicluster-data/ts-0/wal
uuid: "6609f2fb140445708643ec0ce81dbdbf"
format_stamp: "Formatted at 2026-08-07 22:57:11 on dist-test-slave-z8x9"
server_key: "76de60eb0896075eb6c37b91ed50ef6f"
server_key_iv: "150296e9a7c9c964fb6f73a515f0c3eb"
server_key_version: "encryptionkey@0"
I20260807 22:57:11.945647 5678 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786143361779964-25976-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786143361779964-25976-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786143361779964-25976-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:57:11.973857 5678 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:57:11.978138 5678 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:57:11.978453 5678 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:57:11.981763 5678 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:57:11.981885 5678 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:57:11.982074 5678 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:57:11.982162 5678 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 07 22:57:11 dist-test-slave-z8x9 krb5kdc[5473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143431, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.94.1@KRBTEST.COM for oryx/127.25.94.62@KRBTEST.COM
I20260807 22:57:12.008000 5722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:11.982561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.1:41093 (local address 127.25.94.62:34919)
0807 22:57:11.983059 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:11.983066 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:11.984414 (+ 1348us) server_negotiation.cc:407] Connection header received
0807 22:57:11.987664 (+ 3250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:11.987671 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:11.987798 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:11.987939 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:11.990367 (+ 2428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:11.991475 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:11.994068 (+ 2593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:11.994488 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:11.994635 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:11.999614 (+ 4979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:57:11.999626 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:57:11.999629 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:57:11.999669 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:57:12.002963 (+ 3294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:12.003988 (+ 1025us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:12.003994 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:12.003997 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:12.004125 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:12.004800 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:12.004806 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:12.004811 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:12.005446 (+ 635us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:57:12.005637 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0807 22:57:12.007403 (+ 1766us) server_negotiation.cc:299] Negotiation successful
0807 22:57:12.007698 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":268,"threads_started":1}
I20260807 22:57:12.009842 5723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:11.983239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:34919 (local address 127.25.94.1:41093)
0807 22:57:11.984208 (+ 969us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:11.984270 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0807 22:57:11.987264 (+ 2994us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:57:11.988199 (+ 935us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:57:11.988214 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:57:11.989057 (+ 843us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:57:11.990145 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:11.990179 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:11.991651 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:11.991683 (+ 32us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:11.993866 (+ 2183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:11.993879 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:11.994738 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:11.994748 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:11.994980 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:11.995747 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:57:11.995771 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:57:11.999239 (+ 3468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:57:12.003134 (+ 3895us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:57:12.003144 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:57:12.003163 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:57:12.003858 (+ 695us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:57:12.004257 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:57:12.004262 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:57:12.004266 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:57:12.004640 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:57:12.005613 (+ 973us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:57:12.005622 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:57:12.006226 (+ 604us) client_negotiation.cc:769] Sending connection context
0807 22:57:12.006588 (+ 362us) client_negotiation.cc:240] Negotiation successful
0807 22:57:12.007399 (+ 811us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":654,"thread_start_us":382,"threads_started":1}
I20260807 22:57:12.034246 5678 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.1:44883
I20260807 22:57:12.034307 5911 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.1:44883 every 8 connection(s)
I20260807 22:57:12.035382 5722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:12.017994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.1:39463 (local address 127.25.94.62:34919)
0807 22:57:12.018217 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:12.018225 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:12.019169 (+ 944us) server_negotiation.cc:407] Connection header received
0807 22:57:12.019409 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:12.019414 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:12.019498 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:12.019623 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:12.021521 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:12.022576 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:12.025028 (+ 2452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:12.025655 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:12.025816 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:12.028447 (+ 2631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:57:12.028453 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:57:12.028456 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:57:12.028490 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:57:12.031982 (+ 3492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:12.032722 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:12.032730 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:12.032734 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:12.032880 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:12.033405 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:12.033411 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:12.033414 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:12.033716 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:57:12.033850 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0807 22:57:12.034726 (+ 876us) server_negotiation.cc:299] Negotiation successful
0807 22:57:12.035001 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260807 22:57:12.035481 5857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:12.018320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:34919 (local address 127.25.94.1:39463)
0807 22:57:12.019048 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:12.019065 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 22:57:12.019254 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:57:12.019852 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:57:12.019865 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:57:12.020759 (+ 894us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:57:12.021293 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:12.021316 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:12.022768 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:12.022773 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:12.024808 (+ 2035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:12.024833 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:12.026070 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:12.026075 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:12.026203 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:12.026906 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:57:12.026923 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:57:12.028183 (+ 1260us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:57:12.032177 (+ 3994us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:57:12.032181 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:57:12.032184 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:57:12.032588 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:57:12.032998 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:57:12.033003 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:57:12.033008 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:57:12.033234 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:57:12.033989 (+ 755us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:57:12.033997 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:57:12.034344 (+ 347us) client_negotiation.cc:769] Sending connection context
0807 22:57:12.034605 (+ 261us) client_negotiation.cc:240] Negotiation successful
0807 22:57:12.035020 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"thread_start_us":480,"threads_started":1}
I20260807 22:57:12.036275 5678 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786143361779964-25976-0/minicluster-data/ts-0/data/info.pb
I20260807 22:57:12.038784 5912 heartbeater.cc:347] Connected to a master server at 127.25.94.62:34919
I20260807 22:57:12.039419 5912 heartbeater.cc:464] Registering TS with master...
I20260807 22:57:12.040109 5912 heartbeater.cc:511] Master 127.25.94.62:34919 requested a full tablet report, sending...
I20260807 22:57:12.041499 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 5678
I20260807 22:57:12.041730 25976 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786143361779964-25976-0/minicluster-data/ts-0/wal/instance
I20260807 22:57:12.042598 25976 external_mini_cluster.cc:1428] Setting key 5cf44ac122bc2d749ce951bbc77ac545
I20260807 22:57:12.042601 5560 ts_manager.cc:194] Registered new tserver with Master: 6609f2fb140445708643ec0ce81dbdbf (127.25.94.1:44883)
I20260807 22:57:12.044358 5560 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.25.94.1@KRBTEST.COM'} at 127.25.94.1:41093
WARNING: no policy specified for oryx/127.25.94.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.25.94.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.25.94.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786143361779964-25976-0/krb5kdc/oryx_127.25.94.2.keytab.
Entry for principal oryx/127.25.94.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786143361779964-25976-0/krb5kdc/oryx_127.25.94.2.keytab.
I20260807 22:57:12.101625 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786143361779964-25976-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786143361779964-25976-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786143361779964-25976-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786143361779964-25976-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.2:0
--local_ip_for_outbound_sockets=127.25.94.2
--webserver_interface=127.25.94.2
--webserver_port=0
--tserver_master_addrs=127.25.94.62:34919
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786143361779964-25976-0/krb5kdc/oryx_127.25.94.2.keytab
--principal=oryx/127.25.94.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.20:39623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786143361779964-25976-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786143361779964-25976-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786143361779964-25976-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:57:12.452217 5918 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:57:12.452558 5918 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:57:12.452730 5918 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:57:12.460055 5918 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:57:12.460187 5918 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:57:12.460346 5918 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260807 22:57:12.460482 5918 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.2
I20260807 22:57:12.470679 5918 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:39623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786143361779964-25976-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786143361779964-25976-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786143361779964-25976-0/krb5kdc/oryx_127.25.94.2.keytab
--principal=oryx/127.25.94.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786143361779964-25976-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.94.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.25.94.62:34919
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.2
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786143361779964-25976-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 22:57:12.471951 5918 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:57:12.473376 5918 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:57:12.487010 5926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:57:12.486903 5930 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:57:12.487480 5925 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:57:12.488603 5918 server_base.cc:1034] running on GCE node
I20260807 22:57:12.489424 5918 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:57:12.490446 5918 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:57:12.491725 5918 hybrid_clock.cc:648] HybridClock initialized: now 1786143432491655 us; error 49 us; skew 500 ppm
I20260807 22:57:12.492060 5918 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 22:57:12 dist-test-slave-z8x9 krb5kdc[5473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143432, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.94.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:57:12.495564 5918 init.cc:377] Logged in from keytab as oryx/127.25.94.2@KRBTEST.COM (short username oryx)
I20260807 22:57:12.497584 5918 webserver.cc:460] Webserver started at http://127.25.94.2:42633/ using document root <none> and password file <none>
I20260807 22:57:12.498288 5918 fs_manager.cc:359] Metadata directory not provided
I20260807 22:57:12.498389 5918 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:57:12.498672 5918 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:57:12.501688 5918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786143361779964-25976-0/minicluster-data/ts-1/data/instance:
uuid: "29a7220b19354d949c943c99359b54ce"
format_stamp: "Formatted at 2026-08-07 22:57:12 on dist-test-slave-z8x9"
server_key: "2ff1a7bf244d18c98e79bab571ae9273"
server_key_iv: "1ca50f03b976c8524c3642bb9891c1dc"
server_key_version: "encryptionkey@0"
I20260807 22:57:12.502456 5918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786143361779964-25976-0/minicluster-data/ts-1/wal/instance:
uuid: "29a7220b19354d949c943c99359b54ce"
format_stamp: "Formatted at 2026-08-07 22:57:12 on dist-test-slave-z8x9"
server_key: "2ff1a7bf244d18c98e79bab571ae9273"
server_key_iv: "1ca50f03b976c8524c3642bb9891c1dc"
server_key_version: "encryptionkey@0"
I20260807 22:57:12.507885 5918 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260807 22:57:12.511699 5943 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:57:12.513288 5918 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 22:57:12.513439 5918 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786143361779964-25976-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786143361779964-25976-0/minicluster-data/ts-1/wal
uuid: "29a7220b19354d949c943c99359b54ce"
format_stamp: "Formatted at 2026-08-07 22:57:12 on dist-test-slave-z8x9"
server_key: "2ff1a7bf244d18c98e79bab571ae9273"
server_key_iv: "1ca50f03b976c8524c3642bb9891c1dc"
server_key_version: "encryptionkey@0"
I20260807 22:57:12.513671 5918 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786143361779964-25976-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786143361779964-25976-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786143361779964-25976-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:57:12.535393 5918 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:57:12.539810 5918 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:57:12.540055 5918 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:57:12.542893 5918 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:57:12.542986 5918 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:57:12.543226 5918 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:57:12.543318 5918 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 07 22:57:12 dist-test-slave-z8x9 krb5kdc[5473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143432, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.94.2@KRBTEST.COM for oryx/127.25.94.62@KRBTEST.COM
I20260807 22:57:12.568817 5962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:12.543323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.2:59369 (local address 127.25.94.62:34919)
0807 22:57:12.543824 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:12.543830 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:12.545120 (+ 1290us) server_negotiation.cc:407] Connection header received
0807 22:57:12.547510 (+ 2390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:12.547517 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:12.547631 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:12.547753 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:12.550395 (+ 2642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:12.552025 (+ 1630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:12.553985 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:12.554767 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:12.554948 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:12.559798 (+ 4850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:57:12.559809 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:57:12.559818 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:57:12.559886 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:57:12.564326 (+ 4440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:12.565429 (+ 1103us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:12.565437 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:12.565441 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:12.565650 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:12.566355 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:12.566362 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:12.566367 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:12.566675 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:57:12.566886 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0807 22:57:12.568081 (+ 1195us) server_negotiation.cc:299] Negotiation successful
0807 22:57:12.568365 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":249,"threads_started":1}
I20260807 22:57:12.570657 5965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:12.543990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:34919 (local address 127.25.94.2:59369)
0807 22:57:12.544909 (+ 919us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:12.544965 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0807 22:57:12.547203 (+ 2238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:57:12.548055 (+ 852us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:57:12.548070 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:57:12.549016 (+ 946us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:57:12.550102 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:12.550140 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:12.552208 (+ 2068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:12.552212 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:12.553773 (+ 1561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:12.553794 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:12.555099 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:12.555110 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:12.555315 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:12.556132 (+ 817us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:57:12.556160 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:57:12.559496 (+ 3336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:57:12.564502 (+ 5006us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:57:12.564509 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:57:12.564527 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:57:12.565260 (+ 733us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:57:12.565797 (+ 537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:57:12.565809 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:57:12.565819 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:57:12.566193 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:57:12.566849 (+ 656us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:57:12.566862 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:57:12.567485 (+ 623us) client_negotiation.cc:769] Sending connection context
0807 22:57:12.567920 (+ 435us) client_negotiation.cc:240] Negotiation successful
0807 22:57:12.568676 (+ 756us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":642,"thread_start_us":390,"threads_started":1}
I20260807 22:57:12.596240 5962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:12.578613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.2:33385 (local address 127.25.94.62:34919)
0807 22:57:12.578841 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:12.578849 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:12.579337 (+ 488us) server_negotiation.cc:407] Connection header received
0807 22:57:12.579561 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:12.579566 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:12.579642 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:12.579744 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:12.580761 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:12.581593 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:12.584533 (+ 2940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:12.585105 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:12.585487 (+ 382us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:12.588453 (+ 2966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:57:12.588461 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:57:12.588465 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:57:12.588500 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:57:12.592325 (+ 3825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:12.593549 (+ 1224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:12.593557 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:12.593562 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:12.593740 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:12.594324 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:12.594330 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:12.594336 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:12.594938 (+ 602us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:57:12.595118 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0807 22:57:12.595634 (+ 516us) server_negotiation.cc:299] Negotiation successful
0807 22:57:12.595878 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260807 22:57:12.596403 6090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:12.578557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:34919 (local address 127.25.94.2:33385)
0807 22:57:12.579216 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:12.579233 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 22:57:12.579425 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:57:12.579776 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:57:12.579780 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:57:12.580307 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:57:12.580604 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:12.580614 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:12.581769 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:12.581775 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:12.584229 (+ 2454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:12.584255 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:12.585562 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:12.585572 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:12.585843 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:12.586649 (+ 806us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:57:12.586685 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:57:12.588178 (+ 1493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:57:12.592533 (+ 4355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:57:12.592541 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:57:12.592548 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:57:12.593366 (+ 818us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:57:12.593957 (+ 591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:57:12.593961 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:57:12.593963 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:57:12.594177 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:57:12.595127 (+ 950us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:57:12.595132 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:57:12.595360 (+ 228us) client_negotiation.cc:769] Sending connection context
0807 22:57:12.595537 (+ 177us) client_negotiation.cc:240] Negotiation successful
0807 22:57:12.595905 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":386,"threads_started":1}
I20260807 22:57:12.601225 5918 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.2:38743
I20260807 22:57:12.601284 6146 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.2:38743 every 8 connection(s)
I20260807 22:57:12.602578 5918 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786143361779964-25976-0/minicluster-data/ts-1/data/info.pb
I20260807 22:57:12.604651 6147 heartbeater.cc:347] Connected to a master server at 127.25.94.62:34919
I20260807 22:57:12.604912 6147 heartbeater.cc:464] Registering TS with master...
I20260807 22:57:12.605597 6147 heartbeater.cc:511] Master 127.25.94.62:34919 requested a full tablet report, sending...
I20260807 22:57:12.606894 5560 ts_manager.cc:194] Registered new tserver with Master: 29a7220b19354d949c943c99359b54ce (127.25.94.2:38743)
I20260807 22:57:12.607748 5560 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.25.94.2@KRBTEST.COM'} at 127.25.94.2:59369
I20260807 22:57:12.609579 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 5918
I20260807 22:57:12.609752 25976 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786143361779964-25976-0/minicluster-data/ts-1/wal/instance
I20260807 22:57:12.610199 25976 external_mini_cluster.cc:1428] Setting key 05db8d950e6732e3a453909f5b84b859
WARNING: no policy specified for oryx/127.25.94.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.25.94.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.25.94.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786143361779964-25976-0/krb5kdc/oryx_127.25.94.3.keytab.
Entry for principal oryx/127.25.94.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786143361779964-25976-0/krb5kdc/oryx_127.25.94.3.keytab.
I20260807 22:57:12.665414 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786143361779964-25976-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786143361779964-25976-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786143361779964-25976-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786143361779964-25976-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.3:0
--local_ip_for_outbound_sockets=127.25.94.3
--webserver_interface=127.25.94.3
--webserver_port=0
--tserver_master_addrs=127.25.94.62:34919
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786143361779964-25976-0/krb5kdc/oryx_127.25.94.3.keytab
--principal=oryx/127.25.94.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.20:39623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786143361779964-25976-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786143361779964-25976-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786143361779964-25976-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:57:13.007717 6154 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:57:13.008010 6154 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:57:13.008180 6154 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:57:13.016206 6154 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:57:13.016326 6154 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:57:13.016458 6154 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260807 22:57:13.016559 6154 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.3
I20260807 22:57:13.026847 6154 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:39623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786143361779964-25976-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786143361779964-25976-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786143361779964-25976-0/krb5kdc/oryx_127.25.94.3.keytab
--principal=oryx/127.25.94.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786143361779964-25976-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.94.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.25.94.62:34919
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.3
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786143361779964-25976-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 22:57:13.028188 6154 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:57:13.029747 6154 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:57:13.044137 6165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:57:13.044122 6164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:57:13.044341 6168 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:57:13.044993 6154 server_base.cc:1034] running on GCE node
I20260807 22:57:13.045759 6154 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:57:13.046767 6154 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:57:13.047988 6154 hybrid_clock.cc:648] HybridClock initialized: now 1786143433047934 us; error 42 us; skew 500 ppm
I20260807 22:57:13.048049 5912 heartbeater.cc:503] Master 127.25.94.62:34919 was elected leader, sending a full tablet report...
I20260807 22:57:13.048344 6154 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 22:57:13 dist-test-slave-z8x9 krb5kdc[5473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143433, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.94.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:57:13.052000 6154 init.cc:377] Logged in from keytab as oryx/127.25.94.3@KRBTEST.COM (short username oryx)
I20260807 22:57:13.053565 6154 webserver.cc:460] Webserver started at http://127.25.94.3:38585/ using document root <none> and password file <none>
I20260807 22:57:13.054280 6154 fs_manager.cc:359] Metadata directory not provided
I20260807 22:57:13.054420 6154 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:57:13.054770 6154 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:57:13.057777 6154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786143361779964-25976-0/minicluster-data/ts-2/data/instance:
uuid: "7a7eafb4de9948cf855043890af6dfab"
format_stamp: "Formatted at 2026-08-07 22:57:13 on dist-test-slave-z8x9"
server_key: "0faaf981911baf2f3c480dc6c57389ea"
server_key_iv: "64923d41b66f0e40daf6dbcf71940598"
server_key_version: "encryptionkey@0"
I20260807 22:57:13.058599 6154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786143361779964-25976-0/minicluster-data/ts-2/wal/instance:
uuid: "7a7eafb4de9948cf855043890af6dfab"
format_stamp: "Formatted at 2026-08-07 22:57:13 on dist-test-slave-z8x9"
server_key: "0faaf981911baf2f3c480dc6c57389ea"
server_key_iv: "64923d41b66f0e40daf6dbcf71940598"
server_key_version: "encryptionkey@0"
I20260807 22:57:13.064191 6154 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260807 22:57:13.068277 6179 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:57:13.069762 6154 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 22:57:13.069965 6154 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786143361779964-25976-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786143361779964-25976-0/minicluster-data/ts-2/wal
uuid: "7a7eafb4de9948cf855043890af6dfab"
format_stamp: "Formatted at 2026-08-07 22:57:13 on dist-test-slave-z8x9"
server_key: "0faaf981911baf2f3c480dc6c57389ea"
server_key_iv: "64923d41b66f0e40daf6dbcf71940598"
server_key_version: "encryptionkey@0"
I20260807 22:57:13.070221 6154 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786143361779964-25976-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786143361779964-25976-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786143361779964-25976-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:57:13.082904 6154 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:57:13.087750 6154 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:57:13.088104 6154 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:57:13.091338 6154 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:57:13.091466 6154 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:57:13.091686 6154 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:57:13.091820 6154 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Aug 07 22:57:13 dist-test-slave-z8x9 krb5kdc[5473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143433, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.94.3@KRBTEST.COM for oryx/127.25.94.62@KRBTEST.COM
I20260807 22:57:13.118655 5962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:13.091941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.3:34263 (local address 127.25.94.62:34919)
0807 22:57:13.092162 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:13.092169 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:13.093855 (+ 1686us) server_negotiation.cc:407] Connection header received
0807 22:57:13.096002 (+ 2147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:13.096012 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:13.096136 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:13.096256 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:13.099081 (+ 2825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:13.100624 (+ 1543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:13.104381 (+ 3757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:13.105229 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:13.105410 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:13.109755 (+ 4345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:57:13.109765 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:57:13.109774 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:57:13.109832 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:57:13.114107 (+ 4275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:13.115388 (+ 1281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:13.115394 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:13.115396 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:13.115620 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:13.116270 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:13.116276 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:13.116279 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:13.116670 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:57:13.116875 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0807 22:57:13.118009 (+ 1134us) server_negotiation.cc:299] Negotiation successful
0807 22:57:13.118268 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260807 22:57:13.119913 6190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:13.092603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:34919 (local address 127.25.94.3:34263)
0807 22:57:13.093638 (+ 1035us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:13.093698 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0807 22:57:13.095681 (+ 1983us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:57:13.096552 (+ 871us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:57:13.096570 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:57:13.097922 (+ 1352us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:57:13.098833 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:13.098857 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:13.100867 (+ 2010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:13.100873 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:13.104070 (+ 3197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:13.104114 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:13.105573 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:13.105588 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:13.105818 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:13.106643 (+ 825us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:57:13.106667 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:57:13.109452 (+ 2785us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:57:13.114279 (+ 4827us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:57:13.114294 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:57:13.114324 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:57:13.115165 (+ 841us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:57:13.115794 (+ 629us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:57:13.115800 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:57:13.115804 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:57:13.116110 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:57:13.116829 (+ 719us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:57:13.116838 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:57:13.117450 (+ 612us) client_negotiation.cc:769] Sending connection context
0807 22:57:13.117879 (+ 429us) client_negotiation.cc:240] Negotiation successful
0807 22:57:13.118541 (+ 662us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":725,"thread_start_us":448,"threads_started":1}
I20260807 22:57:13.146396 5962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:13.126431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.3:39975 (local address 127.25.94.62:34919)
0807 22:57:13.126600 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:13.126604 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:13.127380 (+ 776us) server_negotiation.cc:407] Connection header received
0807 22:57:13.127635 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:13.127644 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:13.127772 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:13.127901 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:13.129503 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:13.130279 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:13.132875 (+ 2596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:13.133495 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:13.133601 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:13.136842 (+ 3241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:57:13.136848 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:57:13.136852 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:57:13.136893 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:57:13.141342 (+ 4449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:13.142752 (+ 1410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:13.142762 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:13.142769 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:13.143020 (+ 251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:13.144130 (+ 1110us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:13.144142 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:13.144148 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:13.144609 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:57:13.144796 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0807 22:57:13.145520 (+ 724us) server_negotiation.cc:299] Negotiation successful
0807 22:57:13.145892 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260807 22:57:13.146600 6274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:13.126520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:34919 (local address 127.25.94.3:39975)
0807 22:57:13.127252 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:13.127269 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 22:57:13.127476 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:57:13.127961 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:57:13.127973 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:57:13.128772 (+ 799us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:57:13.129308 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:13.129326 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:13.130436 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:13.130445 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:13.132669 (+ 2224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:13.132695 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:13.133722 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:13.133733 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:13.134036 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:13.134969 (+ 933us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:57:13.135004 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:57:13.136493 (+ 1489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:57:13.141539 (+ 5046us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:57:13.141545 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:57:13.141556 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:57:13.142558 (+ 1002us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:57:13.143196 (+ 638us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:57:13.143204 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:57:13.143210 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:57:13.143952 (+ 742us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:57:13.144780 (+ 828us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:57:13.144786 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:57:13.145090 (+ 304us) client_negotiation.cc:769] Sending connection context
0807 22:57:13.145393 (+ 303us) client_negotiation.cc:240] Negotiation successful
0807 22:57:13.145914 (+ 521us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":402,"threads_started":1}
I20260807 22:57:13.158061 6154 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.3:43907
I20260807 22:57:13.158095 6364 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.3:43907 every 8 connection(s)
I20260807 22:57:13.159479 6154 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786143361779964-25976-0/minicluster-data/ts-2/data/info.pb
I20260807 22:57:13.162293 6365 heartbeater.cc:347] Connected to a master server at 127.25.94.62:34919
I20260807 22:57:13.162673 6365 heartbeater.cc:464] Registering TS with master...
I20260807 22:57:13.163556 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 6154
I20260807 22:57:13.163604 6365 heartbeater.cc:511] Master 127.25.94.62:34919 requested a full tablet report, sending...
I20260807 22:57:13.163753 25976 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786143361779964-25976-0/minicluster-data/ts-2/wal/instance
I20260807 22:57:13.164352 25976 external_mini_cluster.cc:1428] Setting key 2580d3abbb318505166227ecef59a3c0
I20260807 22:57:13.165603 5560 ts_manager.cc:194] Registered new tserver with Master: 7a7eafb4de9948cf855043890af6dfab (127.25.94.3:43907)
I20260807 22:57:13.167184 5560 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.25.94.3@KRBTEST.COM'} at 127.25.94.3:34263
I20260807 22:57:13.167943 25976 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 07 22:57:13 dist-test-slave-z8x9 krb5kdc[5473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143433, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 07 22:57:13 dist-test-slave-z8x9 krb5kdc[5473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.25.94.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 22:57:13 dist-test-slave-z8x9 krb5kdc[5473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.25.94.62@KRBTEST.COM, Server not found in Kerberos database
W20260807 22:57:13.232026 5962 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:57:13.220550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36678 (local address 127.25.94.62:34919)
0807 22:57:13.220787 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:13.220794 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:13.220985 (+ 191us) server_negotiation.cc:407] Connection header received
0807 22:57:13.221275 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:13.221289 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:13.221358 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:13.221486 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:13.222917 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:13.223660 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:13.224959 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:13.225431 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:13.225569 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:13.231603 (+ 6034us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36678: BlockingRecv error: recv got EOF from 127.0.0.1:36678 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Aug 07 22:57:13 dist-test-slave-z8x9 krb5kdc[5473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143433, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.25.94.62@KRBTEST.COM
I20260807 22:57:13.252658 5962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:13.237001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36680 (local address 127.25.94.62:34919)
0807 22:57:13.238082 (+ 1081us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:13.238087 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:13.238105 (+ 18us) server_negotiation.cc:407] Connection header received
0807 22:57:13.238161 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:13.238164 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:13.238227 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:13.238352 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:13.239689 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:13.240477 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:13.241910 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:13.242364 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:13.242527 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:13.246827 (+ 4300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:57:13.246833 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:57:13.246837 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:57:13.246888 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:57:13.249767 (+ 2879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:13.250565 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:13.250570 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:13.250572 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:13.250720 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:13.251138 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:13.251142 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:13.251144 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:13.251437 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:57:13.251728 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0807 22:57:13.252050 (+ 322us) server_negotiation.cc:299] Negotiation successful
0807 22:57:13.252291 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":962}
I20260807 22:57:13.257011 5560 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36680:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 22:57:13.264133 5560 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 22:57:13.288332 6404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:13.277457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.2:38743 (local address 127.0.0.1:35548)
0807 22:57:13.278219 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:13.278262 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0807 22:57:13.278446 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:57:13.279763 (+ 1317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:57:13.279772 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:57:13.279802 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:57:13.280333 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:13.280356 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:13.282521 (+ 2165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:13.282528 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:13.285246 (+ 2718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:13.285262 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:13.287223 (+ 1961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:13.287229 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:13.287434 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:13.287503 (+ 69us) client_negotiation.cc:769] Sending connection context
0807 22:57:13.287642 (+ 139us) client_negotiation.cc:240] Negotiation successful
0807 22:57:13.287862 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"mutex_wait_us":42,"thread_start_us":324,"threads_started":1}
I20260807 22:57:13.288513 6406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:13.277628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35548 (local address 127.25.94.2:38743)
0807 22:57:13.279066 (+ 1438us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:13.279075 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:13.279103 (+ 28us) server_negotiation.cc:407] Connection header received
0807 22:57:13.279209 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:13.279219 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:13.279524 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:13.280579 (+ 1055us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:57:13.280801 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:13.282359 (+ 1558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:13.285459 (+ 3100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:13.286970 (+ 1511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:13.287143 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:13.287364 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0807 22:57:13.287953 (+ 589us) server_negotiation.cc:299] Negotiation successful
0807 22:57:13.288156 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1274,"thread_start_us":1118,"threads_started":1}
I20260807 22:57:13.290253 6408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:13.278818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.3:43907 (local address 127.0.0.1:50446)
0807 22:57:13.280733 (+ 1915us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:13.280752 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 22:57:13.280881 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:57:13.281890 (+ 1009us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:57:13.281895 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:57:13.281921 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:57:13.282263 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:13.282276 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:13.284484 (+ 2208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:13.284490 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:13.286754 (+ 2264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:13.286770 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:13.289308 (+ 2538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:13.289316 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:13.289519 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:13.289555 (+ 36us) client_negotiation.cc:769] Sending connection context
0807 22:57:13.289645 (+ 90us) client_negotiation.cc:240] Negotiation successful
0807 22:57:13.289786 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1745,"mutex_wait_us":1,"thread_start_us":1582,"threads_started":1}
I20260807 22:57:13.290810 6410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:13.279069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50446 (local address 127.25.94.3:43907)
0807 22:57:13.279890 (+ 821us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:13.279902 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:13.281009 (+ 1107us) server_negotiation.cc:407] Connection header received
0807 22:57:13.281119 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:13.281126 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:13.281400 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:13.281579 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:57:13.282541 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:13.284229 (+ 1688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:13.286943 (+ 2714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:13.288762 (+ 1819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:13.288880 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:13.289123 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0807 22:57:13.290244 (+ 1121us) server_negotiation.cc:299] Negotiation successful
0807 22:57:13.290449 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":595,"thread_start_us":364,"threads_started":1}
I20260807 22:57:13.293157 6035 tablet_service.cc:1467] Processing CreateTablet for tablet 79ecc93e39d64ce9b42cd4932ea14d9d (DEFAULT_TABLE table=test-table [id=dd7830f23fb1468a912e078271d16b34]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:57:13.294327 6255 tablet_service.cc:1467] Processing CreateTablet for tablet 79ecc93e39d64ce9b42cd4932ea14d9d (DEFAULT_TABLE table=test-table [id=dd7830f23fb1468a912e078271d16b34]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:57:13.295027 6035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79ecc93e39d64ce9b42cd4932ea14d9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:57:13.295946 6255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79ecc93e39d64ce9b42cd4932ea14d9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:57:13.295992 6411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:13.280768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.1:44883 (local address 127.0.0.1:54806)
0807 22:57:13.286789 (+ 6021us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:13.286806 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 22:57:13.286957 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:57:13.287657 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:57:13.287662 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:57:13.287690 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:57:13.288102 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:13.288115 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:13.290045 (+ 1930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:13.290050 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:13.292479 (+ 2429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:13.292498 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:13.294943 (+ 2445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:13.294951 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:13.295289 (+ 338us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:13.295331 (+ 42us) client_negotiation.cc:769] Sending connection context
0807 22:57:13.295449 (+ 118us) client_negotiation.cc:240] Negotiation successful
0807 22:57:13.295613 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5897,"thread_start_us":5732,"threads_started":1}
I20260807 22:57:13.296006 6413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:13.281344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54806 (local address 127.25.94.1:44883)
0807 22:57:13.286289 (+ 4945us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:13.286303 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:13.286962 (+ 659us) server_negotiation.cc:407] Connection header received
0807 22:57:13.287122 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:13.287131 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:13.287439 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:13.287660 (+ 221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:57:13.288307 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:13.289844 (+ 1537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:13.292715 (+ 2871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:13.294250 (+ 1535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:13.294398 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:13.294761 (+ 363us) server_negotiation.cc:1035] Waiting for connection context
0807 22:57:13.295510 (+ 749us) server_negotiation.cc:299] Negotiation successful
0807 22:57:13.295697 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4756,"thread_start_us":4617,"threads_started":1}
I20260807 22:57:13.301000 5785 tablet_service.cc:1467] Processing CreateTablet for tablet 79ecc93e39d64ce9b42cd4932ea14d9d (DEFAULT_TABLE table=test-table [id=dd7830f23fb1468a912e078271d16b34]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:57:13.302840 5785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79ecc93e39d64ce9b42cd4932ea14d9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:57:13.308516 6415 tablet_bootstrap.cc:492] T 79ecc93e39d64ce9b42cd4932ea14d9d P 29a7220b19354d949c943c99359b54ce: Bootstrap starting.
I20260807 22:57:13.311125 6416 tablet_bootstrap.cc:492] T 79ecc93e39d64ce9b42cd4932ea14d9d P 7a7eafb4de9948cf855043890af6dfab: Bootstrap starting.
I20260807 22:57:13.313760 6415 tablet_bootstrap.cc:654] T 79ecc93e39d64ce9b42cd4932ea14d9d P 29a7220b19354d949c943c99359b54ce: Neither blocks nor log segments found. Creating new log.
I20260807 22:57:13.315097 6416 tablet_bootstrap.cc:654] T 79ecc93e39d64ce9b42cd4932ea14d9d P 7a7eafb4de9948cf855043890af6dfab: Neither blocks nor log segments found. Creating new log.
I20260807 22:57:13.315353 6415 log.cc:824] T 79ecc93e39d64ce9b42cd4932ea14d9d P 29a7220b19354d949c943c99359b54ce: Log is configured to *not* fsync() on all Append() calls
I20260807 22:57:13.316450 6416 log.cc:824] T 79ecc93e39d64ce9b42cd4932ea14d9d P 7a7eafb4de9948cf855043890af6dfab: Log is configured to *not* fsync() on all Append() calls
I20260807 22:57:13.318373 6422 tablet_bootstrap.cc:492] T 79ecc93e39d64ce9b42cd4932ea14d9d P 6609f2fb140445708643ec0ce81dbdbf: Bootstrap starting.
I20260807 22:57:13.320174 6416 tablet_bootstrap.cc:492] T 79ecc93e39d64ce9b42cd4932ea14d9d P 7a7eafb4de9948cf855043890af6dfab: No bootstrap required, opened a new log
I20260807 22:57:13.320530 6416 ts_tablet_manager.cc:1397] T 79ecc93e39d64ce9b42cd4932ea14d9d P 7a7eafb4de9948cf855043890af6dfab: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260807 22:57:13.321805 6415 tablet_bootstrap.cc:492] T 79ecc93e39d64ce9b42cd4932ea14d9d P 29a7220b19354d949c943c99359b54ce: No bootstrap required, opened a new log
I20260807 22:57:13.322090 6415 ts_tablet_manager.cc:1397] T 79ecc93e39d64ce9b42cd4932ea14d9d P 29a7220b19354d949c943c99359b54ce: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.005s
I20260807 22:57:13.322322 6422 tablet_bootstrap.cc:654] T 79ecc93e39d64ce9b42cd4932ea14d9d P 6609f2fb140445708643ec0ce81dbdbf: Neither blocks nor log segments found. Creating new log.
I20260807 22:57:13.323457 6422 log.cc:824] T 79ecc93e39d64ce9b42cd4932ea14d9d P 6609f2fb140445708643ec0ce81dbdbf: Log is configured to *not* fsync() on all Append() calls
I20260807 22:57:13.326156 6422 tablet_bootstrap.cc:492] T 79ecc93e39d64ce9b42cd4932ea14d9d P 6609f2fb140445708643ec0ce81dbdbf: No bootstrap required, opened a new log
I20260807 22:57:13.326440 6422 ts_tablet_manager.cc:1397] T 79ecc93e39d64ce9b42cd4932ea14d9d P 6609f2fb140445708643ec0ce81dbdbf: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260807 22:57:13.331921 6416 raft_consensus.cc:348] T 79ecc93e39d64ce9b42cd4932ea14d9d P 7a7eafb4de9948cf855043890af6dfab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29a7220b19354d949c943c99359b54ce" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 38743 } } peers { permanent_uuid: "7a7eafb4de9948cf855043890af6dfab" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 43907 } } peers { permanent_uuid: "6609f2fb140445708643ec0ce81dbdbf" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44883 } }
I20260807 22:57:13.332440 6416 raft_consensus.cc:374] T 79ecc93e39d64ce9b42cd4932ea14d9d P 7a7eafb4de9948cf855043890af6dfab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:57:13.332613 6416 raft_consensus.cc:729] T 79ecc93e39d64ce9b42cd4932ea14d9d P 7a7eafb4de9948cf855043890af6dfab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a7eafb4de9948cf855043890af6dfab, State: Initialized, Role: FOLLOWER
I20260807 22:57:13.332918 6415 raft_consensus.cc:348] T 79ecc93e39d64ce9b42cd4932ea14d9d P 29a7220b19354d949c943c99359b54ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29a7220b19354d949c943c99359b54ce" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 38743 } } peers { permanent_uuid: "7a7eafb4de9948cf855043890af6dfab" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 43907 } } peers { permanent_uuid: "6609f2fb140445708643ec0ce81dbdbf" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44883 } }
I20260807 22:57:13.333431 6415 raft_consensus.cc:374] T 79ecc93e39d64ce9b42cd4932ea14d9d P 29a7220b19354d949c943c99359b54ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:57:13.333537 6415 raft_consensus.cc:729] T 79ecc93e39d64ce9b42cd4932ea14d9d P 29a7220b19354d949c943c99359b54ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29a7220b19354d949c943c99359b54ce, State: Initialized, Role: FOLLOWER
I20260807 22:57:13.333432 6416 consensus_queue.cc:260] T 79ecc93e39d64ce9b42cd4932ea14d9d P 7a7eafb4de9948cf855043890af6dfab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29a7220b19354d949c943c99359b54ce" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 38743 } } peers { permanent_uuid: "7a7eafb4de9948cf855043890af6dfab" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 43907 } } peers { permanent_uuid: "6609f2fb140445708643ec0ce81dbdbf" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44883 } }
I20260807 22:57:13.334614 6415 consensus_queue.cc:260] T 79ecc93e39d64ce9b42cd4932ea14d9d P 29a7220b19354d949c943c99359b54ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29a7220b19354d949c943c99359b54ce" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 38743 } } peers { permanent_uuid: "7a7eafb4de9948cf855043890af6dfab" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 43907 } } peers { permanent_uuid: "6609f2fb140445708643ec0ce81dbdbf" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44883 } }
I20260807 22:57:13.335388 6365 heartbeater.cc:503] Master 127.25.94.62:34919 was elected leader, sending a full tablet report...
I20260807 22:57:13.335703 6422 raft_consensus.cc:348] T 79ecc93e39d64ce9b42cd4932ea14d9d P 6609f2fb140445708643ec0ce81dbdbf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29a7220b19354d949c943c99359b54ce" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 38743 } } peers { permanent_uuid: "7a7eafb4de9948cf855043890af6dfab" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 43907 } } peers { permanent_uuid: "6609f2fb140445708643ec0ce81dbdbf" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44883 } }
I20260807 22:57:13.336030 6147 heartbeater.cc:503] Master 127.25.94.62:34919 was elected leader, sending a full tablet report...
I20260807 22:57:13.336192 6422 raft_consensus.cc:374] T 79ecc93e39d64ce9b42cd4932ea14d9d P 6609f2fb140445708643ec0ce81dbdbf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:57:13.336297 6422 raft_consensus.cc:729] T 79ecc93e39d64ce9b42cd4932ea14d9d P 6609f2fb140445708643ec0ce81dbdbf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6609f2fb140445708643ec0ce81dbdbf, State: Initialized, Role: FOLLOWER
I20260807 22:57:13.336748 6415 ts_tablet_manager.cc:1428] T 79ecc93e39d64ce9b42cd4932ea14d9d P 29a7220b19354d949c943c99359b54ce: Time spent starting tablet: real 0.015s user 0.004s sys 0.012s
I20260807 22:57:13.337204 6422 consensus_queue.cc:260] T 79ecc93e39d64ce9b42cd4932ea14d9d P 6609f2fb140445708643ec0ce81dbdbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29a7220b19354d949c943c99359b54ce" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 38743 } } peers { permanent_uuid: "7a7eafb4de9948cf855043890af6dfab" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 43907 } } peers { permanent_uuid: "6609f2fb140445708643ec0ce81dbdbf" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44883 } }
I20260807 22:57:13.338199 6416 ts_tablet_manager.cc:1428] T 79ecc93e39d64ce9b42cd4932ea14d9d P 7a7eafb4de9948cf855043890af6dfab: Time spent starting tablet: real 0.018s user 0.011s sys 0.003s
I20260807 22:57:13.345849 6422 ts_tablet_manager.cc:1428] T 79ecc93e39d64ce9b42cd4932ea14d9d P 6609f2fb140445708643ec0ce81dbdbf: Time spent starting tablet: real 0.019s user 0.010s sys 0.008s
W20260807 22:57:13.354698 6148 tablet.cc:2369] T 79ecc93e39d64ce9b42cd4932ea14d9d P 29a7220b19354d949c943c99359b54ce: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 22:57:13.411363 6367 tablet.cc:2369] T 79ecc93e39d64ce9b42cd4932ea14d9d P 7a7eafb4de9948cf855043890af6dfab: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 22:57:13.443830 6427 raft_consensus.cc:482] T 79ecc93e39d64ce9b42cd4932ea14d9d P 29a7220b19354d949c943c99359b54ce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 22:57:13.444202 6427 raft_consensus.cc:504] T 79ecc93e39d64ce9b42cd4932ea14d9d P 29a7220b19354d949c943c99359b54ce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29a7220b19354d949c943c99359b54ce" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 38743 } } peers { permanent_uuid: "7a7eafb4de9948cf855043890af6dfab" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 43907 } } peers { permanent_uuid: "6609f2fb140445708643ec0ce81dbdbf" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44883 } }
I20260807 22:57:13.445994 6427 leader_election.cc:288] T 79ecc93e39d64ce9b42cd4932ea14d9d P 29a7220b19354d949c943c99359b54ce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a7eafb4de9948cf855043890af6dfab (127.25.94.3:43907), 6609f2fb140445708643ec0ce81dbdbf (127.25.94.1:44883)
I20260807 22:57:13.453438 6410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:13.446636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.2:57881 (local address 127.25.94.3:43907)
0807 22:57:13.446823 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:13.446827 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:13.447317 (+ 490us) server_negotiation.cc:407] Connection header received
0807 22:57:13.447594 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:13.447604 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:13.447698 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:13.447767 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:57:13.448449 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:13.449580 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:13.451582 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:13.452375 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:13.452541 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:13.452664 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0807 22:57:13.452988 (+ 324us) server_negotiation.cc:299] Negotiation successful
0807 22:57:13.453119 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260807 22:57:13.453598 6431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:13.446440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.3:43907 (local address 127.25.94.2:57881)
0807 22:57:13.447196 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:13.447218 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 22:57:13.447361 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:57:13.447984 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:57:13.447987 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:57:13.448017 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:57:13.448307 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:13.448314 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:13.449699 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:13.449703 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:13.451387 (+ 1684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:13.451403 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:13.452613 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:13.452619 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:13.452792 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:13.452817 (+ 25us) client_negotiation.cc:769] Sending connection context
0807 22:57:13.452959 (+ 142us) client_negotiation.cc:240] Negotiation successful
0807 22:57:13.453097 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":624,"mutex_wait_us":41,"thread_start_us":401,"threads_started":1}
I20260807 22:57:13.453686 6413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:13.447566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.2:57839 (local address 127.25.94.1:44883)
0807 22:57:13.447793 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:13.447815 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:13.448051 (+ 236us) server_negotiation.cc:407] Connection header received
0807 22:57:13.448278 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:13.448283 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:13.448382 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:13.448483 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:57:13.448992 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:13.450062 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:13.451914 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:13.452690 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:13.452876 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:13.452984 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0807 22:57:13.453300 (+ 316us) server_negotiation.cc:299] Negotiation successful
0807 22:57:13.453427 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260807 22:57:13.453882 6432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:13.447324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.1:44883 (local address 127.25.94.2:57839)
0807 22:57:13.447942 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:13.447956 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 22:57:13.448138 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:57:13.448503 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:57:13.448506 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:57:13.448524 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:57:13.448829 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:13.448839 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:13.450200 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:13.450206 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:13.451748 (+ 1542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:13.451762 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:13.452941 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:13.452946 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:13.453115 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:13.453138 (+ 23us) client_negotiation.cc:769] Sending connection context
0807 22:57:13.453267 (+ 129us) client_negotiation.cc:240] Negotiation successful
0807 22:57:13.453405 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"mutex_wait_us":22,"thread_start_us":296,"threads_started":1}
I20260807 22:57:13.455008 5820 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79ecc93e39d64ce9b42cd4932ea14d9d" candidate_uuid: "29a7220b19354d949c943c99359b54ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6609f2fb140445708643ec0ce81dbdbf" is_pre_election: true
I20260807 22:57:13.455008 6292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79ecc93e39d64ce9b42cd4932ea14d9d" candidate_uuid: "29a7220b19354d949c943c99359b54ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a7eafb4de9948cf855043890af6dfab" is_pre_election: true
I20260807 22:57:13.455619 6292 raft_consensus.cc:2454] T 79ecc93e39d64ce9b42cd4932ea14d9d P 7a7eafb4de9948cf855043890af6dfab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29a7220b19354d949c943c99359b54ce in term 0.
I20260807 22:57:13.455619 5820 raft_consensus.cc:2454] T 79ecc93e39d64ce9b42cd4932ea14d9d P 6609f2fb140445708643ec0ce81dbdbf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29a7220b19354d949c943c99359b54ce in term 0.
I20260807 22:57:13.456298 5946 leader_election.cc:302] T 79ecc93e39d64ce9b42cd4932ea14d9d P 29a7220b19354d949c943c99359b54ce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29a7220b19354d949c943c99359b54ce, 7a7eafb4de9948cf855043890af6dfab; no voters:
I20260807 22:57:13.456635 6427 raft_consensus.cc:2781] T 79ecc93e39d64ce9b42cd4932ea14d9d P 29a7220b19354d949c943c99359b54ce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 22:57:13.456904 6427 raft_consensus.cc:482] T 79ecc93e39d64ce9b42cd4932ea14d9d P 29a7220b19354d949c943c99359b54ce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 22:57:13.457033 6427 raft_consensus.cc:3037] T 79ecc93e39d64ce9b42cd4932ea14d9d P 29a7220b19354d949c943c99359b54ce [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:57:13.459223 6427 raft_consensus.cc:504] T 79ecc93e39d64ce9b42cd4932ea14d9d P 29a7220b19354d949c943c99359b54ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29a7220b19354d949c943c99359b54ce" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 38743 } } peers { permanent_uuid: "7a7eafb4de9948cf855043890af6dfab" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 43907 } } peers { permanent_uuid: "6609f2fb140445708643ec0ce81dbdbf" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44883 } }
I20260807 22:57:13.459794 6427 leader_election.cc:288] T 79ecc93e39d64ce9b42cd4932ea14d9d P 29a7220b19354d949c943c99359b54ce [CANDIDATE]: Term 1 election: Requested vote from peers 7a7eafb4de9948cf855043890af6dfab (127.25.94.3:43907), 6609f2fb140445708643ec0ce81dbdbf (127.25.94.1:44883)
I20260807 22:57:13.460286 6292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79ecc93e39d64ce9b42cd4932ea14d9d" candidate_uuid: "29a7220b19354d949c943c99359b54ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a7eafb4de9948cf855043890af6dfab"
I20260807 22:57:13.460379 5820 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79ecc93e39d64ce9b42cd4932ea14d9d" candidate_uuid: "29a7220b19354d949c943c99359b54ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6609f2fb140445708643ec0ce81dbdbf"
I20260807 22:57:13.460505 6292 raft_consensus.cc:3037] T 79ecc93e39d64ce9b42cd4932ea14d9d P 7a7eafb4de9948cf855043890af6dfab [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:57:13.460574 5820 raft_consensus.cc:3037] T 79ecc93e39d64ce9b42cd4932ea14d9d P 6609f2fb140445708643ec0ce81dbdbf [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:57:13.463754 5820 raft_consensus.cc:2454] T 79ecc93e39d64ce9b42cd4932ea14d9d P 6609f2fb140445708643ec0ce81dbdbf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29a7220b19354d949c943c99359b54ce in term 1.
I20260807 22:57:13.463754 6292 raft_consensus.cc:2454] T 79ecc93e39d64ce9b42cd4932ea14d9d P 7a7eafb4de9948cf855043890af6dfab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29a7220b19354d949c943c99359b54ce in term 1.
I20260807 22:57:13.464140 5946 leader_election.cc:302] T 79ecc93e39d64ce9b42cd4932ea14d9d P 29a7220b19354d949c943c99359b54ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29a7220b19354d949c943c99359b54ce, 7a7eafb4de9948cf855043890af6dfab; no voters:
I20260807 22:57:13.464432 6427 raft_consensus.cc:2781] T 79ecc93e39d64ce9b42cd4932ea14d9d P 29a7220b19354d949c943c99359b54ce [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:57:13.464843 6427 raft_consensus.cc:686] T 79ecc93e39d64ce9b42cd4932ea14d9d P 29a7220b19354d949c943c99359b54ce [term 1 LEADER]: Becoming Leader. State: Replica: 29a7220b19354d949c943c99359b54ce, State: Running, Role: LEADER
I20260807 22:57:13.465406 6427 consensus_queue.cc:237] T 79ecc93e39d64ce9b42cd4932ea14d9d P 29a7220b19354d949c943c99359b54ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29a7220b19354d949c943c99359b54ce" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 38743 } } peers { permanent_uuid: "7a7eafb4de9948cf855043890af6dfab" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 43907 } } peers { permanent_uuid: "6609f2fb140445708643ec0ce81dbdbf" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44883 } }
I20260807 22:57:13.469784 5560 catalog_manager.cc:5495] T 79ecc93e39d64ce9b42cd4932ea14d9d P 29a7220b19354d949c943c99359b54ce reported cstate change: term changed from 0 to 1, leader changed from <none> to 29a7220b19354d949c943c99359b54ce (127.25.94.2). New cstate: current_term: 1 leader_uuid: "29a7220b19354d949c943c99359b54ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29a7220b19354d949c943c99359b54ce" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 38743 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a7eafb4de9948cf855043890af6dfab" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 43907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6609f2fb140445708643ec0ce81dbdbf" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44883 } health_report { overall_health: UNKNOWN } } }
I20260807 22:57:13.529613 6406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:13.524532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35560 (local address 127.25.94.2:38743)
0807 22:57:13.524712 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:13.524717 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:13.524736 (+ 19us) server_negotiation.cc:407] Connection header received
0807 22:57:13.524938 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:13.524942 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:13.525021 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:13.525162 (+ 141us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 22:57:13.525643 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:13.526354 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:13.527599 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:13.528010 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:13.528102 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:13.528505 (+ 403us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 22:57:13.529046 (+ 541us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 22:57:13.529153 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0807 22:57:13.529324 (+ 171us) server_negotiation.cc:299] Negotiation successful
0807 22:57:13.529393 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260807 22:57:13.538509 5820 raft_consensus.cc:1261] T 79ecc93e39d64ce9b42cd4932ea14d9d P 6609f2fb140445708643ec0ce81dbdbf [term 1 FOLLOWER]: Refusing update from remote peer 29a7220b19354d949c943c99359b54ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 22:57:13.538508 6292 raft_consensus.cc:1261] T 79ecc93e39d64ce9b42cd4932ea14d9d P 7a7eafb4de9948cf855043890af6dfab [term 1 FOLLOWER]: Refusing update from remote peer 29a7220b19354d949c943c99359b54ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 22:57:13.539543 6433 consensus_queue.cc:1035] T 79ecc93e39d64ce9b42cd4932ea14d9d P 29a7220b19354d949c943c99359b54ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "6609f2fb140445708643ec0ce81dbdbf" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 22:57:13.539937 6427 consensus_queue.cc:1035] T 79ecc93e39d64ce9b42cd4932ea14d9d P 29a7220b19354d949c943c99359b54ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a7eafb4de9948cf855043890af6dfab" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 43907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260807 22:57:13.540067 5914 tablet.cc:2369] T 79ecc93e39d64ce9b42cd4932ea14d9d P 6609f2fb140445708643ec0ce81dbdbf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 22:57:13.555962 6440 mvcc.cc:205] Tried to move back new op lower bound from 7316043503762784256 to 7316043503478259712. Current Snapshot: MvccSnapshot[applied={T|T < 7316043503762784256}]
I20260807 22:57:13.565717 5560 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36680:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 22:57:13.566035 5560 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36680:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 22:57:13.570036 5560 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1f81ab1e409a4b718bf275a1e92f4448: Sending DeleteTablet for 3 replicas of tablet 79ecc93e39d64ce9b42cd4932ea14d9d
I20260807 22:57:13.571249 6035 tablet_service.cc:1514] Processing DeleteTablet for tablet 79ecc93e39d64ce9b42cd4932ea14d9d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:57:13 UTC) from {username='oryx', principal='oryx/127.25.94.62@KRBTEST.COM'} at 127.0.0.1:35548
I20260807 22:57:13.571449 5785 tablet_service.cc:1514] Processing DeleteTablet for tablet 79ecc93e39d64ce9b42cd4932ea14d9d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:57:13 UTC) from {username='oryx', principal='oryx/127.25.94.62@KRBTEST.COM'} at 127.0.0.1:54806
I20260807 22:57:13.572424 6255 tablet_service.cc:1514] Processing DeleteTablet for tablet 79ecc93e39d64ce9b42cd4932ea14d9d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:57:13 UTC) from {username='oryx', principal='oryx/127.25.94.62@KRBTEST.COM'} at 127.0.0.1:50446
I20260807 22:57:13.572536 5560 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36680:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 22:57:13.573329 5560 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 22:57:13.575997 6453 tablet_replica.cc:330] T 79ecc93e39d64ce9b42cd4932ea14d9d P 7a7eafb4de9948cf855043890af6dfab: stopping tablet replica
I20260807 22:57:13.576534 6453 raft_consensus.cc:2229] T 79ecc93e39d64ce9b42cd4932ea14d9d P 7a7eafb4de9948cf855043890af6dfab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 22:57:13.577194 6453 raft_consensus.cc:2258] T 79ecc93e39d64ce9b42cd4932ea14d9d P 7a7eafb4de9948cf855043890af6dfab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 22:57:13.577775 6451 tablet_replica.cc:330] T 79ecc93e39d64ce9b42cd4932ea14d9d P 6609f2fb140445708643ec0ce81dbdbf: stopping tablet replica
I20260807 22:57:13.578039 6450 tablet_replica.cc:330] T 79ecc93e39d64ce9b42cd4932ea14d9d P 29a7220b19354d949c943c99359b54ce: stopping tablet replica
I20260807 22:57:13.578478 6450 raft_consensus.cc:2229] T 79ecc93e39d64ce9b42cd4932ea14d9d P 29a7220b19354d949c943c99359b54ce [term 1 LEADER]: Raft consensus shutting down.
I20260807 22:57:13.578496 6451 raft_consensus.cc:2229] T 79ecc93e39d64ce9b42cd4932ea14d9d P 6609f2fb140445708643ec0ce81dbdbf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 22:57:13.578974 6451 raft_consensus.cc:2258] T 79ecc93e39d64ce9b42cd4932ea14d9d P 6609f2fb140445708643ec0ce81dbdbf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 22:57:13.579201 6450 raft_consensus.cc:2258] T 79ecc93e39d64ce9b42cd4932ea14d9d P 29a7220b19354d949c943c99359b54ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 22:57:13.581317 6450 ts_tablet_manager.cc:1905] T 79ecc93e39d64ce9b42cd4932ea14d9d P 29a7220b19354d949c943c99359b54ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 22:57:13.581779 5785 tablet_service.cc:1467] Processing CreateTablet for tablet 3928efd33c5d4ca3b84760be76bdbcb8 (DEFAULT_TABLE table=test-table [id=2b65045699864f58b95d14cfaccf28e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:57:13.582252 5785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3928efd33c5d4ca3b84760be76bdbcb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:57:13.582470 6255 tablet_service.cc:1467] Processing CreateTablet for tablet 3928efd33c5d4ca3b84760be76bdbcb8 (DEFAULT_TABLE table=test-table [id=2b65045699864f58b95d14cfaccf28e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:57:13.582871 6255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3928efd33c5d4ca3b84760be76bdbcb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:57:13.588732 6422 tablet_bootstrap.cc:492] T 3928efd33c5d4ca3b84760be76bdbcb8 P 6609f2fb140445708643ec0ce81dbdbf: Bootstrap starting.
I20260807 22:57:13.588876 6450 ts_tablet_manager.cc:1918] T 79ecc93e39d64ce9b42cd4932ea14d9d P 29a7220b19354d949c943c99359b54ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 22:57:13.589049 6450 log.cc:1196] T 79ecc93e39d64ce9b42cd4932ea14d9d P 29a7220b19354d949c943c99359b54ce: Deleting WAL directory at /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786143361779964-25976-0/minicluster-data/ts-1/wal/wals/79ecc93e39d64ce9b42cd4932ea14d9d
I20260807 22:57:13.589856 6035 tablet_service.cc:1467] Processing CreateTablet for tablet 3928efd33c5d4ca3b84760be76bdbcb8 (DEFAULT_TABLE table=test-table [id=2b65045699864f58b95d14cfaccf28e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:57:13.590258 6035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3928efd33c5d4ca3b84760be76bdbcb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:57:13.590260 6453 ts_tablet_manager.cc:1905] T 79ecc93e39d64ce9b42cd4932ea14d9d P 7a7eafb4de9948cf855043890af6dfab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 22:57:13.590528 6422 tablet_bootstrap.cc:654] T 3928efd33c5d4ca3b84760be76bdbcb8 P 6609f2fb140445708643ec0ce81dbdbf: Neither blocks nor log segments found. Creating new log.
I20260807 22:57:13.590974 6451 ts_tablet_manager.cc:1905] T 79ecc93e39d64ce9b42cd4932ea14d9d P 6609f2fb140445708643ec0ce81dbdbf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 22:57:13.597069 6453 ts_tablet_manager.cc:1918] T 79ecc93e39d64ce9b42cd4932ea14d9d P 7a7eafb4de9948cf855043890af6dfab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 22:57:13.597282 6453 log.cc:1196] T 79ecc93e39d64ce9b42cd4932ea14d9d P 7a7eafb4de9948cf855043890af6dfab: Deleting WAL directory at /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786143361779964-25976-0/minicluster-data/ts-2/wal/wals/79ecc93e39d64ce9b42cd4932ea14d9d
I20260807 22:57:13.597823 6453 ts_tablet_manager.cc:1939] T 79ecc93e39d64ce9b42cd4932ea14d9d P 7a7eafb4de9948cf855043890af6dfab: Deleting consensus metadata
I20260807 22:57:13.598486 6416 tablet_bootstrap.cc:492] T 3928efd33c5d4ca3b84760be76bdbcb8 P 7a7eafb4de9948cf855043890af6dfab: Bootstrap starting.
I20260807 22:57:13.598623 6450 ts_tablet_manager.cc:1939] T 79ecc93e39d64ce9b42cd4932ea14d9d P 29a7220b19354d949c943c99359b54ce: Deleting consensus metadata
I20260807 22:57:13.600384 5536 catalog_manager.cc:4841] TS 29a7220b19354d949c943c99359b54ce (127.25.94.2:38743): tablet 79ecc93e39d64ce9b42cd4932ea14d9d (table test-table [id=dd7830f23fb1468a912e078271d16b34]) successfully deleted
I20260807 22:57:13.600594 6416 tablet_bootstrap.cc:654] T 3928efd33c5d4ca3b84760be76bdbcb8 P 7a7eafb4de9948cf855043890af6dfab: Neither blocks nor log segments found. Creating new log.
I20260807 22:57:13.597931 6451 ts_tablet_manager.cc:1918] T 79ecc93e39d64ce9b42cd4932ea14d9d P 6609f2fb140445708643ec0ce81dbdbf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 22:57:13.601397 6451 log.cc:1196] T 79ecc93e39d64ce9b42cd4932ea14d9d P 6609f2fb140445708643ec0ce81dbdbf: Deleting WAL directory at /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786143361779964-25976-0/minicluster-data/ts-0/wal/wals/79ecc93e39d64ce9b42cd4932ea14d9d
I20260807 22:57:13.602010 6451 ts_tablet_manager.cc:1939] T 79ecc93e39d64ce9b42cd4932ea14d9d P 6609f2fb140445708643ec0ce81dbdbf: Deleting consensus metadata
I20260807 22:57:13.604578 6415 tablet_bootstrap.cc:492] T 3928efd33c5d4ca3b84760be76bdbcb8 P 29a7220b19354d949c943c99359b54ce: Bootstrap starting.
I20260807 22:57:13.605967 5538 catalog_manager.cc:4841] TS 6609f2fb140445708643ec0ce81dbdbf (127.25.94.1:44883): tablet 79ecc93e39d64ce9b42cd4932ea14d9d (table test-table [id=dd7830f23fb1468a912e078271d16b34]) successfully deleted
I20260807 22:57:13.606208 5538 catalog_manager.cc:4841] TS 7a7eafb4de9948cf855043890af6dfab (127.25.94.3:43907): tablet 79ecc93e39d64ce9b42cd4932ea14d9d (table test-table [id=dd7830f23fb1468a912e078271d16b34]) successfully deleted
I20260807 22:57:13.606390 6415 tablet_bootstrap.cc:654] T 3928efd33c5d4ca3b84760be76bdbcb8 P 29a7220b19354d949c943c99359b54ce: Neither blocks nor log segments found. Creating new log.
I20260807 22:57:13.606876 6422 tablet_bootstrap.cc:492] T 3928efd33c5d4ca3b84760be76bdbcb8 P 6609f2fb140445708643ec0ce81dbdbf: No bootstrap required, opened a new log
I20260807 22:57:13.607043 6422 ts_tablet_manager.cc:1397] T 3928efd33c5d4ca3b84760be76bdbcb8 P 6609f2fb140445708643ec0ce81dbdbf: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.006s
I20260807 22:57:13.607872 6422 raft_consensus.cc:348] T 3928efd33c5d4ca3b84760be76bdbcb8 P 6609f2fb140445708643ec0ce81dbdbf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6609f2fb140445708643ec0ce81dbdbf" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44883 } } peers { permanent_uuid: "29a7220b19354d949c943c99359b54ce" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 38743 } } peers { permanent_uuid: "7a7eafb4de9948cf855043890af6dfab" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 43907 } }
I20260807 22:57:13.608119 6422 raft_consensus.cc:374] T 3928efd33c5d4ca3b84760be76bdbcb8 P 6609f2fb140445708643ec0ce81dbdbf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:57:13.608215 6422 raft_consensus.cc:729] T 3928efd33c5d4ca3b84760be76bdbcb8 P 6609f2fb140445708643ec0ce81dbdbf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6609f2fb140445708643ec0ce81dbdbf, State: Initialized, Role: FOLLOWER
I20260807 22:57:13.608511 6422 consensus_queue.cc:260] T 3928efd33c5d4ca3b84760be76bdbcb8 P 6609f2fb140445708643ec0ce81dbdbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6609f2fb140445708643ec0ce81dbdbf" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44883 } } peers { permanent_uuid: "29a7220b19354d949c943c99359b54ce" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 38743 } } peers { permanent_uuid: "7a7eafb4de9948cf855043890af6dfab" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 43907 } }
I20260807 22:57:13.609303 6422 ts_tablet_manager.cc:1428] T 3928efd33c5d4ca3b84760be76bdbcb8 P 6609f2fb140445708643ec0ce81dbdbf: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260807 22:57:13.610931 6416 tablet_bootstrap.cc:492] T 3928efd33c5d4ca3b84760be76bdbcb8 P 7a7eafb4de9948cf855043890af6dfab: No bootstrap required, opened a new log
I20260807 22:57:13.611129 6416 ts_tablet_manager.cc:1397] T 3928efd33c5d4ca3b84760be76bdbcb8 P 7a7eafb4de9948cf855043890af6dfab: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.005s
I20260807 22:57:13.611982 6416 raft_consensus.cc:348] T 3928efd33c5d4ca3b84760be76bdbcb8 P 7a7eafb4de9948cf855043890af6dfab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6609f2fb140445708643ec0ce81dbdbf" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44883 } } peers { permanent_uuid: "29a7220b19354d949c943c99359b54ce" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 38743 } } peers { permanent_uuid: "7a7eafb4de9948cf855043890af6dfab" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 43907 } }
I20260807 22:57:13.612257 6416 raft_consensus.cc:374] T 3928efd33c5d4ca3b84760be76bdbcb8 P 7a7eafb4de9948cf855043890af6dfab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:57:13.612404 6416 raft_consensus.cc:729] T 3928efd33c5d4ca3b84760be76bdbcb8 P 7a7eafb4de9948cf855043890af6dfab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a7eafb4de9948cf855043890af6dfab, State: Initialized, Role: FOLLOWER
I20260807 22:57:13.612813 6416 consensus_queue.cc:260] T 3928efd33c5d4ca3b84760be76bdbcb8 P 7a7eafb4de9948cf855043890af6dfab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6609f2fb140445708643ec0ce81dbdbf" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44883 } } peers { permanent_uuid: "29a7220b19354d949c943c99359b54ce" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 38743 } } peers { permanent_uuid: "7a7eafb4de9948cf855043890af6dfab" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 43907 } }
I20260807 22:57:13.614648 6416 ts_tablet_manager.cc:1428] T 3928efd33c5d4ca3b84760be76bdbcb8 P 7a7eafb4de9948cf855043890af6dfab: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260807 22:57:13.618137 6415 tablet_bootstrap.cc:492] T 3928efd33c5d4ca3b84760be76bdbcb8 P 29a7220b19354d949c943c99359b54ce: No bootstrap required, opened a new log
I20260807 22:57:13.618302 6415 ts_tablet_manager.cc:1397] T 3928efd33c5d4ca3b84760be76bdbcb8 P 29a7220b19354d949c943c99359b54ce: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.005s
I20260807 22:57:13.619122 6415 raft_consensus.cc:348] T 3928efd33c5d4ca3b84760be76bdbcb8 P 29a7220b19354d949c943c99359b54ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6609f2fb140445708643ec0ce81dbdbf" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44883 } } peers { permanent_uuid: "29a7220b19354d949c943c99359b54ce" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 38743 } } peers { permanent_uuid: "7a7eafb4de9948cf855043890af6dfab" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 43907 } }
I20260807 22:57:13.619355 6415 raft_consensus.cc:374] T 3928efd33c5d4ca3b84760be76bdbcb8 P 29a7220b19354d949c943c99359b54ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:57:13.619494 6415 raft_consensus.cc:729] T 3928efd33c5d4ca3b84760be76bdbcb8 P 29a7220b19354d949c943c99359b54ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29a7220b19354d949c943c99359b54ce, State: Initialized, Role: FOLLOWER
I20260807 22:57:13.619735 6415 consensus_queue.cc:260] T 3928efd33c5d4ca3b84760be76bdbcb8 P 29a7220b19354d949c943c99359b54ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6609f2fb140445708643ec0ce81dbdbf" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44883 } } peers { permanent_uuid: "29a7220b19354d949c943c99359b54ce" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 38743 } } peers { permanent_uuid: "7a7eafb4de9948cf855043890af6dfab" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 43907 } }
I20260807 22:57:13.620348 6415 ts_tablet_manager.cc:1428] T 3928efd33c5d4ca3b84760be76bdbcb8 P 29a7220b19354d949c943c99359b54ce: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260807 22:57:13.760958 6429 raft_consensus.cc:482] T 3928efd33c5d4ca3b84760be76bdbcb8 P 6609f2fb140445708643ec0ce81dbdbf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 22:57:13.761263 6429 raft_consensus.cc:504] T 3928efd33c5d4ca3b84760be76bdbcb8 P 6609f2fb140445708643ec0ce81dbdbf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6609f2fb140445708643ec0ce81dbdbf" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44883 } } peers { permanent_uuid: "29a7220b19354d949c943c99359b54ce" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 38743 } } peers { permanent_uuid: "7a7eafb4de9948cf855043890af6dfab" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 43907 } }
I20260807 22:57:13.763480 6429 leader_election.cc:288] T 3928efd33c5d4ca3b84760be76bdbcb8 P 6609f2fb140445708643ec0ce81dbdbf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29a7220b19354d949c943c99359b54ce (127.25.94.2:38743), 7a7eafb4de9948cf855043890af6dfab (127.25.94.3:43907)
I20260807 22:57:13.771726 6410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:13.763943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.1:54903 (local address 127.25.94.3:43907)
0807 22:57:13.764145 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:13.764152 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:13.764976 (+ 824us) server_negotiation.cc:407] Connection header received
0807 22:57:13.765358 (+ 382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:13.765364 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:13.765450 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:13.765544 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:57:13.766303 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:13.767592 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:13.770005 (+ 2413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:13.770793 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:13.770947 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:13.771060 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0807 22:57:13.771353 (+ 293us) server_negotiation.cc:299] Negotiation successful
0807 22:57:13.771477 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260807 22:57:13.771850 6459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:13.763813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.3:43907 (local address 127.25.94.1:54903)
0807 22:57:13.764848 (+ 1035us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:13.764874 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0807 22:57:13.765134 (+ 260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:57:13.765594 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:57:13.765599 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:57:13.765624 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:57:13.766092 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:13.766106 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:13.767753 (+ 1647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:13.767758 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:13.769828 (+ 2070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:13.769840 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:13.771008 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:13.771026 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:13.771189 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:13.771213 (+ 24us) client_negotiation.cc:769] Sending connection context
0807 22:57:13.771317 (+ 104us) client_negotiation.cc:240] Negotiation successful
0807 22:57:13.771444 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":869,"mutex_wait_us":27,"thread_start_us":601,"threads_started":1}
I20260807 22:57:13.772401 6292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3928efd33c5d4ca3b84760be76bdbcb8" candidate_uuid: "6609f2fb140445708643ec0ce81dbdbf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a7eafb4de9948cf855043890af6dfab" is_pre_election: true
I20260807 22:57:13.772722 6292 raft_consensus.cc:2454] T 3928efd33c5d4ca3b84760be76bdbcb8 P 7a7eafb4de9948cf855043890af6dfab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6609f2fb140445708643ec0ce81dbdbf in term 0.
I20260807 22:57:13.773128 6406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:13.763830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.1:54389 (local address 127.25.94.2:38743)
0807 22:57:13.764051 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:13.764061 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:13.765406 (+ 1345us) server_negotiation.cc:407] Connection header received
0807 22:57:13.765575 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:13.765581 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:13.765668 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:13.765774 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:57:13.766452 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:13.767280 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:13.769972 (+ 2692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:13.771440 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:13.771590 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:13.771687 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0807 22:57:13.772614 (+ 927us) server_negotiation.cc:299] Negotiation successful
0807 22:57:13.772792 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260807 22:57:13.773344 5708 leader_election.cc:302] T 3928efd33c5d4ca3b84760be76bdbcb8 P 6609f2fb140445708643ec0ce81dbdbf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6609f2fb140445708643ec0ce81dbdbf, 7a7eafb4de9948cf855043890af6dfab; no voters:
I20260807 22:57:13.773501 6462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:13.764379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.2:38743 (local address 127.25.94.1:54389)
0807 22:57:13.765266 (+ 887us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:13.765286 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 22:57:13.765430 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:57:13.765806 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:57:13.765810 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:57:13.765830 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:57:13.766274 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:13.766285 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:13.767415 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:13.767423 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:13.769788 (+ 2365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:13.769805 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:13.772164 (+ 2359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:13.772172 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:13.772365 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:13.772416 (+ 51us) client_negotiation.cc:769] Sending connection context
0807 22:57:13.772564 (+ 148us) client_negotiation.cc:240] Negotiation successful
0807 22:57:13.772794 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":710,"mutex_wait_us":1,"thread_start_us":506,"threads_started":1}
I20260807 22:57:13.773809 6429 raft_consensus.cc:2781] T 3928efd33c5d4ca3b84760be76bdbcb8 P 6609f2fb140445708643ec0ce81dbdbf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 22:57:13.773953 6429 raft_consensus.cc:482] T 3928efd33c5d4ca3b84760be76bdbcb8 P 6609f2fb140445708643ec0ce81dbdbf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 22:57:13.774099 6429 raft_consensus.cc:3037] T 3928efd33c5d4ca3b84760be76bdbcb8 P 6609f2fb140445708643ec0ce81dbdbf [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:57:13.774883 6085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3928efd33c5d4ca3b84760be76bdbcb8" candidate_uuid: "6609f2fb140445708643ec0ce81dbdbf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29a7220b19354d949c943c99359b54ce" is_pre_election: true
I20260807 22:57:13.775417 6085 raft_consensus.cc:2454] T 3928efd33c5d4ca3b84760be76bdbcb8 P 29a7220b19354d949c943c99359b54ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6609f2fb140445708643ec0ce81dbdbf in term 0.
I20260807 22:57:13.776671 6429 raft_consensus.cc:504] T 3928efd33c5d4ca3b84760be76bdbcb8 P 6609f2fb140445708643ec0ce81dbdbf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6609f2fb140445708643ec0ce81dbdbf" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44883 } } peers { permanent_uuid: "29a7220b19354d949c943c99359b54ce" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 38743 } } peers { permanent_uuid: "7a7eafb4de9948cf855043890af6dfab" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 43907 } }
I20260807 22:57:13.777420 6429 leader_election.cc:288] T 3928efd33c5d4ca3b84760be76bdbcb8 P 6609f2fb140445708643ec0ce81dbdbf [CANDIDATE]: Term 1 election: Requested vote from peers 29a7220b19354d949c943c99359b54ce (127.25.94.2:38743), 7a7eafb4de9948cf855043890af6dfab (127.25.94.3:43907)
I20260807 22:57:13.777779 6085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3928efd33c5d4ca3b84760be76bdbcb8" candidate_uuid: "6609f2fb140445708643ec0ce81dbdbf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29a7220b19354d949c943c99359b54ce"
I20260807 22:57:13.778007 6085 raft_consensus.cc:3037] T 3928efd33c5d4ca3b84760be76bdbcb8 P 29a7220b19354d949c943c99359b54ce [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:57:13.778008 6292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3928efd33c5d4ca3b84760be76bdbcb8" candidate_uuid: "6609f2fb140445708643ec0ce81dbdbf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a7eafb4de9948cf855043890af6dfab"
I20260807 22:57:13.778245 6292 raft_consensus.cc:3037] T 3928efd33c5d4ca3b84760be76bdbcb8 P 7a7eafb4de9948cf855043890af6dfab [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:57:13.780357 6085 raft_consensus.cc:2454] T 3928efd33c5d4ca3b84760be76bdbcb8 P 29a7220b19354d949c943c99359b54ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6609f2fb140445708643ec0ce81dbdbf in term 1.
I20260807 22:57:13.780763 5706 leader_election.cc:302] T 3928efd33c5d4ca3b84760be76bdbcb8 P 6609f2fb140445708643ec0ce81dbdbf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29a7220b19354d949c943c99359b54ce, 6609f2fb140445708643ec0ce81dbdbf; no voters:
I20260807 22:57:13.781085 6429 raft_consensus.cc:2781] T 3928efd33c5d4ca3b84760be76bdbcb8 P 6609f2fb140445708643ec0ce81dbdbf [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:57:13.781508 6292 raft_consensus.cc:2454] T 3928efd33c5d4ca3b84760be76bdbcb8 P 7a7eafb4de9948cf855043890af6dfab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6609f2fb140445708643ec0ce81dbdbf in term 1.
I20260807 22:57:13.781634 6429 raft_consensus.cc:686] T 3928efd33c5d4ca3b84760be76bdbcb8 P 6609f2fb140445708643ec0ce81dbdbf [term 1 LEADER]: Becoming Leader. State: Replica: 6609f2fb140445708643ec0ce81dbdbf, State: Running, Role: LEADER
I20260807 22:57:13.782294 6429 consensus_queue.cc:237] T 3928efd33c5d4ca3b84760be76bdbcb8 P 6609f2fb140445708643ec0ce81dbdbf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6609f2fb140445708643ec0ce81dbdbf" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44883 } } peers { permanent_uuid: "29a7220b19354d949c943c99359b54ce" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 38743 } } peers { permanent_uuid: "7a7eafb4de9948cf855043890af6dfab" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 43907 } }
I20260807 22:57:13.785967 5560 catalog_manager.cc:5495] T 3928efd33c5d4ca3b84760be76bdbcb8 P 6609f2fb140445708643ec0ce81dbdbf reported cstate change: term changed from 0 to 1, leader changed from <none> to 6609f2fb140445708643ec0ce81dbdbf (127.25.94.1). New cstate: current_term: 1 leader_uuid: "6609f2fb140445708643ec0ce81dbdbf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6609f2fb140445708643ec0ce81dbdbf" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44883 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "29a7220b19354d949c943c99359b54ce" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 38743 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a7eafb4de9948cf855043890af6dfab" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 43907 } health_report { overall_health: UNKNOWN } } }
Aug 07 22:57:13 dist-test-slave-z8x9 krb5kdc[5473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143431, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.94.62@KRBTEST.COM for oryx/127.25.94.62@KRBTEST.COM
I20260807 22:57:13.847275 6477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:13.831639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:34919 (local address 127.0.0.1:36682)
0807 22:57:13.832445 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:13.832475 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0807 22:57:13.832658 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:57:13.833076 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:57:13.833082 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:57:13.834127 (+ 1045us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:57:13.834452 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:13.834467 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:13.835719 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:13.835725 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:13.837294 (+ 1569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:13.837312 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:13.838040 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:13.838047 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:13.838233 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:13.839081 (+ 848us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:57:13.839108 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:57:13.841760 (+ 2652us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:57:13.844698 (+ 2938us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:57:13.844710 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:57:13.844717 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:57:13.845134 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:57:13.845542 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:57:13.845548 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:57:13.845551 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:57:13.845697 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:57:13.846206 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:57:13.846214 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:57:13.846376 (+ 162us) client_negotiation.cc:769] Sending connection context
0807 22:57:13.846539 (+ 163us) client_negotiation.cc:240] Negotiation successful
0807 22:57:13.846828 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":596,"thread_start_us":423,"threads_started":1}
I20260807 22:57:13.847319 6478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:13.831705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36682 (local address 127.25.94.62:34919)
0807 22:57:13.832739 (+ 1034us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:13.832744 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:13.832765 (+ 21us) server_negotiation.cc:407] Connection header received
0807 22:57:13.832818 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:13.832821 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:13.832878 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:13.832967 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:13.834638 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:13.835560 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:13.837442 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:13.837853 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:13.837948 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:13.841952 (+ 4004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:57:13.841957 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:57:13.841960 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:57:13.841988 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:57:13.844556 (+ 2568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:13.845296 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:13.845303 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:13.845311 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:13.845432 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:13.845816 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:13.845823 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:13.845830 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:13.846070 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:57:13.846222 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0807 22:57:13.846629 (+ 407us) server_negotiation.cc:299] Negotiation successful
0807 22:57:13.846954 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":134,"server-negotiator.queue_time_us":930,"thread_start_us":787,"threads_started":1}
I20260807 22:57:13.852820 5560 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.25.94.62@KRBTEST.COM'} at 127.0.0.1:36682:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260807 22:57:13.853837 5560 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 22:57:13.860914 5785 tablet_service.cc:1467] Processing CreateTablet for tablet 7cfcb2c73e334a65844bd828373e8005 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b543d4ea01ee4b40a89588237f640f84]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260807 22:57:13.861028 6035 tablet_service.cc:1467] Processing CreateTablet for tablet 7cfcb2c73e334a65844bd828373e8005 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b543d4ea01ee4b40a89588237f640f84]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260807 22:57:13.861619 5785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7cfcb2c73e334a65844bd828373e8005. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:57:13.861653 6035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7cfcb2c73e334a65844bd828373e8005. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:57:13.866219 6255 tablet_service.cc:1467] Processing CreateTablet for tablet 7cfcb2c73e334a65844bd828373e8005 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b543d4ea01ee4b40a89588237f640f84]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260807 22:57:13.866814 6255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7cfcb2c73e334a65844bd828373e8005. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:57:13.867694 6415 tablet_bootstrap.cc:492] T 7cfcb2c73e334a65844bd828373e8005 P 29a7220b19354d949c943c99359b54ce: Bootstrap starting.
I20260807 22:57:13.869010 6422 tablet_bootstrap.cc:492] T 7cfcb2c73e334a65844bd828373e8005 P 6609f2fb140445708643ec0ce81dbdbf: Bootstrap starting.
I20260807 22:57:13.869565 6415 tablet_bootstrap.cc:654] T 7cfcb2c73e334a65844bd828373e8005 P 29a7220b19354d949c943c99359b54ce: Neither blocks nor log segments found. Creating new log.
I20260807 22:57:13.870754 6422 tablet_bootstrap.cc:654] T 7cfcb2c73e334a65844bd828373e8005 P 6609f2fb140445708643ec0ce81dbdbf: Neither blocks nor log segments found. Creating new log.
I20260807 22:57:13.876868 6416 tablet_bootstrap.cc:492] T 7cfcb2c73e334a65844bd828373e8005 P 7a7eafb4de9948cf855043890af6dfab: Bootstrap starting.
I20260807 22:57:13.878783 6416 tablet_bootstrap.cc:654] T 7cfcb2c73e334a65844bd828373e8005 P 7a7eafb4de9948cf855043890af6dfab: Neither blocks nor log segments found. Creating new log.
I20260807 22:57:13.879274 6415 tablet_bootstrap.cc:492] T 7cfcb2c73e334a65844bd828373e8005 P 29a7220b19354d949c943c99359b54ce: No bootstrap required, opened a new log
I20260807 22:57:13.879470 6415 ts_tablet_manager.cc:1397] T 7cfcb2c73e334a65844bd828373e8005 P 29a7220b19354d949c943c99359b54ce: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.002s
I20260807 22:57:13.880532 6415 raft_consensus.cc:348] T 7cfcb2c73e334a65844bd828373e8005 P 29a7220b19354d949c943c99359b54ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6609f2fb140445708643ec0ce81dbdbf" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44883 } } peers { permanent_uuid: "29a7220b19354d949c943c99359b54ce" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 38743 } } peers { permanent_uuid: "7a7eafb4de9948cf855043890af6dfab" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 43907 } }
I20260807 22:57:13.880901 6415 raft_consensus.cc:374] T 7cfcb2c73e334a65844bd828373e8005 P 29a7220b19354d949c943c99359b54ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:57:13.881009 6415 raft_consensus.cc:729] T 7cfcb2c73e334a65844bd828373e8005 P 29a7220b19354d949c943c99359b54ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29a7220b19354d949c943c99359b54ce, State: Initialized, Role: FOLLOWER
I20260807 22:57:13.881438 6415 consensus_queue.cc:260] T 7cfcb2c73e334a65844bd828373e8005 P 29a7220b19354d949c943c99359b54ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6609f2fb140445708643ec0ce81dbdbf" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44883 } } peers { permanent_uuid: "29a7220b19354d949c943c99359b54ce" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 38743 } } peers { permanent_uuid: "7a7eafb4de9948cf855043890af6dfab" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 43907 } }
I20260807 22:57:13.882055 6427 tablet_replica.cc:439] T 7cfcb2c73e334a65844bd828373e8005 P 29a7220b19354d949c943c99359b54ce: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6609f2fb140445708643ec0ce81dbdbf" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44883 } } peers { permanent_uuid: "29a7220b19354d949c943c99359b54ce" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 38743 } } peers { permanent_uuid: "7a7eafb4de9948cf855043890af6dfab" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 43907 } } }
I20260807 22:57:13.882448 6427 tablet_replica.cc:442] T 7cfcb2c73e334a65844bd828373e8005 P 29a7220b19354d949c943c99359b54ce: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 22:57:13.886181 6415 ts_tablet_manager.cc:1428] T 7cfcb2c73e334a65844bd828373e8005 P 29a7220b19354d949c943c99359b54ce: Time spent starting tablet: real 0.007s user 0.000s sys 0.003s
I20260807 22:57:13.887985 6422 tablet_bootstrap.cc:492] T 7cfcb2c73e334a65844bd828373e8005 P 6609f2fb140445708643ec0ce81dbdbf: No bootstrap required, opened a new log
I20260807 22:57:13.888231 6422 ts_tablet_manager.cc:1397] T 7cfcb2c73e334a65844bd828373e8005 P 6609f2fb140445708643ec0ce81dbdbf: Time spent bootstrapping tablet: real 0.019s user 0.008s sys 0.005s
I20260807 22:57:13.889070 6422 raft_consensus.cc:348] T 7cfcb2c73e334a65844bd828373e8005 P 6609f2fb140445708643ec0ce81dbdbf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6609f2fb140445708643ec0ce81dbdbf" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44883 } } peers { permanent_uuid: "29a7220b19354d949c943c99359b54ce" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 38743 } } peers { permanent_uuid: "7a7eafb4de9948cf855043890af6dfab" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 43907 } }
I20260807 22:57:13.889424 6422 raft_consensus.cc:374] T 7cfcb2c73e334a65844bd828373e8005 P 6609f2fb140445708643ec0ce81dbdbf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:57:13.889570 6422 raft_consensus.cc:729] T 7cfcb2c73e334a65844bd828373e8005 P 6609f2fb140445708643ec0ce81dbdbf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6609f2fb140445708643ec0ce81dbdbf, State: Initialized, Role: FOLLOWER
I20260807 22:57:13.889902 6422 consensus_queue.cc:260] T 7cfcb2c73e334a65844bd828373e8005 P 6609f2fb140445708643ec0ce81dbdbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6609f2fb140445708643ec0ce81dbdbf" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44883 } } peers { permanent_uuid: "29a7220b19354d949c943c99359b54ce" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 38743 } } peers { permanent_uuid: "7a7eafb4de9948cf855043890af6dfab" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 43907 } }
I20260807 22:57:13.890432 6429 tablet_replica.cc:439] T 7cfcb2c73e334a65844bd828373e8005 P 6609f2fb140445708643ec0ce81dbdbf: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6609f2fb140445708643ec0ce81dbdbf" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44883 } } peers { permanent_uuid: "29a7220b19354d949c943c99359b54ce" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 38743 } } peers { permanent_uuid: "7a7eafb4de9948cf855043890af6dfab" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 43907 } } }
I20260807 22:57:13.890772 6429 tablet_replica.cc:442] T 7cfcb2c73e334a65844bd828373e8005 P 6609f2fb140445708643ec0ce81dbdbf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 22:57:13.891371 6422 ts_tablet_manager.cc:1428] T 7cfcb2c73e334a65844bd828373e8005 P 6609f2fb140445708643ec0ce81dbdbf: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260807 22:57:13.893203 6416 tablet_bootstrap.cc:492] T 7cfcb2c73e334a65844bd828373e8005 P 7a7eafb4de9948cf855043890af6dfab: No bootstrap required, opened a new log
I20260807 22:57:13.893404 6416 ts_tablet_manager.cc:1397] T 7cfcb2c73e334a65844bd828373e8005 P 7a7eafb4de9948cf855043890af6dfab: Time spent bootstrapping tablet: real 0.017s user 0.012s sys 0.003s
I20260807 22:57:13.894378 6416 raft_consensus.cc:348] T 7cfcb2c73e334a65844bd828373e8005 P 7a7eafb4de9948cf855043890af6dfab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6609f2fb140445708643ec0ce81dbdbf" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44883 } } peers { permanent_uuid: "29a7220b19354d949c943c99359b54ce" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 38743 } } peers { permanent_uuid: "7a7eafb4de9948cf855043890af6dfab" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 43907 } }
I20260807 22:57:13.894654 6416 raft_consensus.cc:374] T 7cfcb2c73e334a65844bd828373e8005 P 7a7eafb4de9948cf855043890af6dfab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:57:13.894752 6416 raft_consensus.cc:729] T 7cfcb2c73e334a65844bd828373e8005 P 7a7eafb4de9948cf855043890af6dfab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a7eafb4de9948cf855043890af6dfab, State: Initialized, Role: FOLLOWER
I20260807 22:57:13.895025 6416 consensus_queue.cc:260] T 7cfcb2c73e334a65844bd828373e8005 P 7a7eafb4de9948cf855043890af6dfab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6609f2fb140445708643ec0ce81dbdbf" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44883 } } peers { permanent_uuid: "29a7220b19354d949c943c99359b54ce" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 38743 } } peers { permanent_uuid: "7a7eafb4de9948cf855043890af6dfab" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 43907 } }
I20260807 22:57:13.895622 6416 ts_tablet_manager.cc:1428] T 7cfcb2c73e334a65844bd828373e8005 P 7a7eafb4de9948cf855043890af6dfab: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260807 22:57:13.895550 6426 tablet_replica.cc:439] T 7cfcb2c73e334a65844bd828373e8005 P 7a7eafb4de9948cf855043890af6dfab: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6609f2fb140445708643ec0ce81dbdbf" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44883 } } peers { permanent_uuid: "29a7220b19354d949c943c99359b54ce" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 38743 } } peers { permanent_uuid: "7a7eafb4de9948cf855043890af6dfab" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 43907 } } }
I20260807 22:57:13.895896 6426 tablet_replica.cc:442] T 7cfcb2c73e334a65844bd828373e8005 P 7a7eafb4de9948cf855043890af6dfab: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 22:57:14.039170 6427 raft_consensus.cc:482] T 7cfcb2c73e334a65844bd828373e8005 P 29a7220b19354d949c943c99359b54ce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 22:57:14.039391 6427 raft_consensus.cc:504] T 7cfcb2c73e334a65844bd828373e8005 P 29a7220b19354d949c943c99359b54ce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6609f2fb140445708643ec0ce81dbdbf" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44883 } } peers { permanent_uuid: "29a7220b19354d949c943c99359b54ce" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 38743 } } peers { permanent_uuid: "7a7eafb4de9948cf855043890af6dfab" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 43907 } }
I20260807 22:57:14.040115 6427 leader_election.cc:288] T 7cfcb2c73e334a65844bd828373e8005 P 29a7220b19354d949c943c99359b54ce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6609f2fb140445708643ec0ce81dbdbf (127.25.94.1:44883), 7a7eafb4de9948cf855043890af6dfab (127.25.94.3:43907)
I20260807 22:57:14.040784 6292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cfcb2c73e334a65844bd828373e8005" candidate_uuid: "29a7220b19354d949c943c99359b54ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a7eafb4de9948cf855043890af6dfab" is_pre_election: true
I20260807 22:57:14.040824 5820 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cfcb2c73e334a65844bd828373e8005" candidate_uuid: "29a7220b19354d949c943c99359b54ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6609f2fb140445708643ec0ce81dbdbf" is_pre_election: true
I20260807 22:57:14.041067 6292 raft_consensus.cc:2454] T 7cfcb2c73e334a65844bd828373e8005 P 7a7eafb4de9948cf855043890af6dfab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29a7220b19354d949c943c99359b54ce in term 0.
I20260807 22:57:14.041093 5820 raft_consensus.cc:2454] T 7cfcb2c73e334a65844bd828373e8005 P 6609f2fb140445708643ec0ce81dbdbf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29a7220b19354d949c943c99359b54ce in term 0.
I20260807 22:57:14.041798 5946 leader_election.cc:302] T 7cfcb2c73e334a65844bd828373e8005 P 29a7220b19354d949c943c99359b54ce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29a7220b19354d949c943c99359b54ce, 6609f2fb140445708643ec0ce81dbdbf; no voters:
I20260807 22:57:14.042291 6427 raft_consensus.cc:2781] T 7cfcb2c73e334a65844bd828373e8005 P 29a7220b19354d949c943c99359b54ce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 22:57:14.042425 6427 raft_consensus.cc:482] T 7cfcb2c73e334a65844bd828373e8005 P 29a7220b19354d949c943c99359b54ce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 22:57:14.042532 6427 raft_consensus.cc:3037] T 7cfcb2c73e334a65844bd828373e8005 P 29a7220b19354d949c943c99359b54ce [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:57:14.045702 6427 raft_consensus.cc:504] T 7cfcb2c73e334a65844bd828373e8005 P 29a7220b19354d949c943c99359b54ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6609f2fb140445708643ec0ce81dbdbf" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44883 } } peers { permanent_uuid: "29a7220b19354d949c943c99359b54ce" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 38743 } } peers { permanent_uuid: "7a7eafb4de9948cf855043890af6dfab" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 43907 } }
I20260807 22:57:14.046332 6427 leader_election.cc:288] T 7cfcb2c73e334a65844bd828373e8005 P 29a7220b19354d949c943c99359b54ce [CANDIDATE]: Term 1 election: Requested vote from peers 6609f2fb140445708643ec0ce81dbdbf (127.25.94.1:44883), 7a7eafb4de9948cf855043890af6dfab (127.25.94.3:43907)
I20260807 22:57:14.046934 5820 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cfcb2c73e334a65844bd828373e8005" candidate_uuid: "29a7220b19354d949c943c99359b54ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6609f2fb140445708643ec0ce81dbdbf"
I20260807 22:57:14.046923 6292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cfcb2c73e334a65844bd828373e8005" candidate_uuid: "29a7220b19354d949c943c99359b54ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a7eafb4de9948cf855043890af6dfab"
I20260807 22:57:14.047153 5820 raft_consensus.cc:3037] T 7cfcb2c73e334a65844bd828373e8005 P 6609f2fb140445708643ec0ce81dbdbf [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:57:14.047168 6292 raft_consensus.cc:3037] T 7cfcb2c73e334a65844bd828373e8005 P 7a7eafb4de9948cf855043890af6dfab [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:57:14.050362 6292 raft_consensus.cc:2454] T 7cfcb2c73e334a65844bd828373e8005 P 7a7eafb4de9948cf855043890af6dfab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29a7220b19354d949c943c99359b54ce in term 1.
I20260807 22:57:14.050429 5820 raft_consensus.cc:2454] T 7cfcb2c73e334a65844bd828373e8005 P 6609f2fb140445708643ec0ce81dbdbf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29a7220b19354d949c943c99359b54ce in term 1.
I20260807 22:57:14.050884 5946 leader_election.cc:302] T 7cfcb2c73e334a65844bd828373e8005 P 29a7220b19354d949c943c99359b54ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29a7220b19354d949c943c99359b54ce, 6609f2fb140445708643ec0ce81dbdbf; no voters:
I20260807 22:57:14.051204 6427 raft_consensus.cc:2781] T 7cfcb2c73e334a65844bd828373e8005 P 29a7220b19354d949c943c99359b54ce [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:57:14.051419 6427 raft_consensus.cc:686] T 7cfcb2c73e334a65844bd828373e8005 P 29a7220b19354d949c943c99359b54ce [term 1 LEADER]: Becoming Leader. State: Replica: 29a7220b19354d949c943c99359b54ce, State: Running, Role: LEADER
I20260807 22:57:14.051749 6427 consensus_queue.cc:237] T 7cfcb2c73e334a65844bd828373e8005 P 29a7220b19354d949c943c99359b54ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6609f2fb140445708643ec0ce81dbdbf" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44883 } } peers { permanent_uuid: "29a7220b19354d949c943c99359b54ce" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 38743 } } peers { permanent_uuid: "7a7eafb4de9948cf855043890af6dfab" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 43907 } }
I20260807 22:57:14.054237 6443 tablet_replica.cc:439] T 7cfcb2c73e334a65844bd828373e8005 P 29a7220b19354d949c943c99359b54ce: TxnStatusTablet state changed. Reason: New leader 29a7220b19354d949c943c99359b54ce. Latest consensus state: current_term: 1 leader_uuid: "29a7220b19354d949c943c99359b54ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6609f2fb140445708643ec0ce81dbdbf" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44883 } } peers { permanent_uuid: "29a7220b19354d949c943c99359b54ce" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 38743 } } peers { permanent_uuid: "7a7eafb4de9948cf855043890af6dfab" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 43907 } } }
I20260807 22:57:14.054526 6443 tablet_replica.cc:442] T 7cfcb2c73e334a65844bd828373e8005 P 29a7220b19354d949c943c99359b54ce: This TxnStatusTablet replica's current role is: LEADER
I20260807 22:57:14.055083 6491 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260807 22:57:14.055323 6491 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260807 22:57:14.055372 5560 catalog_manager.cc:5495] T 7cfcb2c73e334a65844bd828373e8005 P 29a7220b19354d949c943c99359b54ce reported cstate change: term changed from 0 to 1, leader changed from <none> to 29a7220b19354d949c943c99359b54ce (127.25.94.2). New cstate: current_term: 1 leader_uuid: "29a7220b19354d949c943c99359b54ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6609f2fb140445708643ec0ce81dbdbf" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44883 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "29a7220b19354d949c943c99359b54ce" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 38743 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a7eafb4de9948cf855043890af6dfab" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 43907 } health_report { overall_health: UNKNOWN } } }
I20260807 22:57:14.083541 6406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:14.078289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35562 (local address 127.25.94.2:38743)
0807 22:57:14.078638 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:14.078644 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:14.078668 (+ 24us) server_negotiation.cc:407] Connection header received
0807 22:57:14.078892 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:14.078898 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:14.079009 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:14.079133 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 22:57:14.079516 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:14.080280 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:14.081654 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:14.082134 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:14.082280 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:14.082736 (+ 456us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 22:57:14.082869 (+ 133us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 22:57:14.082974 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0807 22:57:14.083138 (+ 164us) server_negotiation.cc:299] Negotiation successful
0807 22:57:14.083220 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227}
I20260807 22:57:14.083585 6477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:14.078255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.2:38743 (local address 127.0.0.1:35562)
0807 22:57:14.078512 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:14.078535 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0807 22:57:14.078683 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:57:14.079168 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:57:14.079173 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:57:14.079187 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0807 22:57:14.079376 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:14.079385 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:14.080417 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:14.080425 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:14.081498 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:14.081511 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:14.082292 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:14.082306 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:14.082473 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:14.082535 (+ 62us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0807 22:57:14.082979 (+ 444us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0807 22:57:14.083003 (+ 24us) client_negotiation.cc:769] Sending connection context
0807 22:57:14.083111 (+ 108us) client_negotiation.cc:240] Negotiation successful
0807 22:57:14.083268 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260807 22:57:14.087769 6292 raft_consensus.cc:1261] T 7cfcb2c73e334a65844bd828373e8005 P 7a7eafb4de9948cf855043890af6dfab [term 1 FOLLOWER]: Refusing update from remote peer 29a7220b19354d949c943c99359b54ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 22:57:14.087824 5820 raft_consensus.cc:1261] T 7cfcb2c73e334a65844bd828373e8005 P 6609f2fb140445708643ec0ce81dbdbf [term 1 FOLLOWER]: Refusing update from remote peer 29a7220b19354d949c943c99359b54ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 22:57:14.088461 6443 consensus_queue.cc:1035] T 7cfcb2c73e334a65844bd828373e8005 P 29a7220b19354d949c943c99359b54ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a7eafb4de9948cf855043890af6dfab" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 43907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 22:57:14.088748 6427 consensus_queue.cc:1035] T 7cfcb2c73e334a65844bd828373e8005 P 29a7220b19354d949c943c99359b54ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "6609f2fb140445708643ec0ce81dbdbf" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 22:57:14.092422 6426 tablet_replica.cc:439] T 7cfcb2c73e334a65844bd828373e8005 P 7a7eafb4de9948cf855043890af6dfab: TxnStatusTablet state changed. Reason: New leader 29a7220b19354d949c943c99359b54ce. Latest consensus state: current_term: 1 leader_uuid: "29a7220b19354d949c943c99359b54ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6609f2fb140445708643ec0ce81dbdbf" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44883 } } peers { permanent_uuid: "29a7220b19354d949c943c99359b54ce" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 38743 } } peers { permanent_uuid: "7a7eafb4de9948cf855043890af6dfab" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 43907 } } }
I20260807 22:57:14.092715 6426 tablet_replica.cc:442] T 7cfcb2c73e334a65844bd828373e8005 P 7a7eafb4de9948cf855043890af6dfab: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 22:57:14.094298 6494 mvcc.cc:205] Tried to move back new op lower bound from 7316043506017251328 to 7316043505878700032. Current Snapshot: MvccSnapshot[applied={T|T < 7316043506017251328}]
I20260807 22:57:14.102384 6429 tablet_replica.cc:439] T 7cfcb2c73e334a65844bd828373e8005 P 6609f2fb140445708643ec0ce81dbdbf: TxnStatusTablet state changed. Reason: New leader 29a7220b19354d949c943c99359b54ce. Latest consensus state: current_term: 1 leader_uuid: "29a7220b19354d949c943c99359b54ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6609f2fb140445708643ec0ce81dbdbf" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44883 } } peers { permanent_uuid: "29a7220b19354d949c943c99359b54ce" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 38743 } } peers { permanent_uuid: "7a7eafb4de9948cf855043890af6dfab" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 43907 } } }
I20260807 22:57:14.103032 6429 tablet_replica.cc:442] T 7cfcb2c73e334a65844bd828373e8005 P 6609f2fb140445708643ec0ce81dbdbf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 22:57:14.105245 6426 tablet_replica.cc:439] T 7cfcb2c73e334a65844bd828373e8005 P 7a7eafb4de9948cf855043890af6dfab: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "29a7220b19354d949c943c99359b54ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6609f2fb140445708643ec0ce81dbdbf" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44883 } } peers { permanent_uuid: "29a7220b19354d949c943c99359b54ce" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 38743 } } peers { permanent_uuid: "7a7eafb4de9948cf855043890af6dfab" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 43907 } } }
I20260807 22:57:14.105664 6426 tablet_replica.cc:442] T 7cfcb2c73e334a65844bd828373e8005 P 7a7eafb4de9948cf855043890af6dfab: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 22:57:14.109035 6427 tablet_replica.cc:439] T 7cfcb2c73e334a65844bd828373e8005 P 29a7220b19354d949c943c99359b54ce: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "29a7220b19354d949c943c99359b54ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6609f2fb140445708643ec0ce81dbdbf" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44883 } } peers { permanent_uuid: "29a7220b19354d949c943c99359b54ce" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 38743 } } peers { permanent_uuid: "7a7eafb4de9948cf855043890af6dfab" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 43907 } } }
I20260807 22:57:14.109443 6427 tablet_replica.cc:442] T 7cfcb2c73e334a65844bd828373e8005 P 29a7220b19354d949c943c99359b54ce: This TxnStatusTablet replica's current role is: LEADER
I20260807 22:57:14.109900 6497 mvcc.cc:205] Tried to move back new op lower bound from 7316043506017251328 to 7316043505878700032. Current Snapshot: MvccSnapshot[applied={T|T < 7316043506017251328}]
I20260807 22:57:14.111159 6504 tablet_replica.cc:439] T 7cfcb2c73e334a65844bd828373e8005 P 29a7220b19354d949c943c99359b54ce: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "29a7220b19354d949c943c99359b54ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6609f2fb140445708643ec0ce81dbdbf" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44883 } } peers { permanent_uuid: "29a7220b19354d949c943c99359b54ce" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 38743 } } peers { permanent_uuid: "7a7eafb4de9948cf855043890af6dfab" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 43907 } } }
I20260807 22:57:14.111620 6504 tablet_replica.cc:442] T 7cfcb2c73e334a65844bd828373e8005 P 29a7220b19354d949c943c99359b54ce: This TxnStatusTablet replica's current role is: LEADER
I20260807 22:57:14.114346 6429 tablet_replica.cc:439] T 7cfcb2c73e334a65844bd828373e8005 P 6609f2fb140445708643ec0ce81dbdbf: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "29a7220b19354d949c943c99359b54ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6609f2fb140445708643ec0ce81dbdbf" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44883 } } peers { permanent_uuid: "29a7220b19354d949c943c99359b54ce" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 38743 } } peers { permanent_uuid: "7a7eafb4de9948cf855043890af6dfab" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 43907 } } }
I20260807 22:57:14.114656 6429 tablet_replica.cc:442] T 7cfcb2c73e334a65844bd828373e8005 P 6609f2fb140445708643ec0ce81dbdbf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 22:57:14.128144 6508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:14.121234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54814 (local address 127.25.94.1:44883)
0807 22:57:14.121874 (+ 640us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:14.121881 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:14.122177 (+ 296us) server_negotiation.cc:407] Connection header received
0807 22:57:14.122469 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:14.122474 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:14.122618 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:14.122897 (+ 279us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 22:57:14.123421 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:14.124301 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:14.125798 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:14.126248 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:14.126401 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:14.126855 (+ 454us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 22:57:14.127450 (+ 595us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 22:57:14.127613 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0807 22:57:14.127695 (+ 82us) server_negotiation.cc:299] Negotiation successful
0807 22:57:14.127785 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":487,"thread_start_us":308,"threads_started":1}
I20260807 22:57:14.146564 6406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:14.140405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.1:57503 (local address 127.25.94.2:38743)
0807 22:57:14.140619 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:14.140623 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:14.141208 (+ 585us) server_negotiation.cc:407] Connection header received
0807 22:57:14.141444 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:14.141448 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:14.141520 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:14.141619 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 22:57:14.142141 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:14.142979 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:14.144474 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:14.145010 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:14.145154 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:14.145722 (+ 568us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 22:57:14.145876 (+ 154us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 22:57:14.145984 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0807 22:57:14.146128 (+ 144us) server_negotiation.cc:299] Negotiation successful
0807 22:57:14.146214 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260807 22:57:14.146646 6512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:14.140426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.2:38743 (local address 127.25.94.1:57503)
0807 22:57:14.141066 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:14.141082 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 22:57:14.141287 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:57:14.141637 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:57:14.141642 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:57:14.141653 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0807 22:57:14.141985 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:14.141994 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:14.143131 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:14.143141 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:14.144306 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:14.144320 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:14.145315 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:14.145325 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:14.145456 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:14.145518 (+ 62us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0807 22:57:14.145987 (+ 469us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0807 22:57:14.146001 (+ 14us) client_negotiation.cc:769] Sending connection context
0807 22:57:14.146104 (+ 103us) client_negotiation.cc:240] Negotiation successful
0807 22:57:14.146250 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":346,"threads_started":1}
I20260807 22:57:14.157805 6292 raft_consensus.cc:1261] T 3928efd33c5d4ca3b84760be76bdbcb8 P 7a7eafb4de9948cf855043890af6dfab [term 1 FOLLOWER]: Refusing update from remote peer 6609f2fb140445708643ec0ce81dbdbf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 22:57:14.158272 6085 raft_consensus.cc:1261] T 3928efd33c5d4ca3b84760be76bdbcb8 P 29a7220b19354d949c943c99359b54ce [term 1 FOLLOWER]: Refusing update from remote peer 6609f2fb140445708643ec0ce81dbdbf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 22:57:14.158737 6463 consensus_queue.cc:1035] T 3928efd33c5d4ca3b84760be76bdbcb8 P 6609f2fb140445708643ec0ce81dbdbf [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a7eafb4de9948cf855043890af6dfab" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 43907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 22:57:14.159133 6429 consensus_queue.cc:1035] T 3928efd33c5d4ca3b84760be76bdbcb8 P 6609f2fb140445708643ec0ce81dbdbf [LEADER]: Connected to new peer: Peer: permanent_uuid: "29a7220b19354d949c943c99359b54ce" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 38743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 22:57:14.220757 6508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:14.211937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.2:56225 (local address 127.25.94.1:44883)
0807 22:57:14.212412 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:14.212422 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:14.214482 (+ 2060us) server_negotiation.cc:407] Connection header received
0807 22:57:14.214720 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:14.214725 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:14.214795 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:14.214880 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 22:57:14.215601 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:14.216656 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:14.218876 (+ 2220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:14.219285 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:14.219441 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:14.219943 (+ 502us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 22:57:14.220059 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 22:57:14.220163 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0807 22:57:14.220317 (+ 154us) server_negotiation.cc:299] Negotiation successful
0807 22:57:14.220413 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284}
I20260807 22:57:14.220901 6519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:14.211099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.1:44883 (local address 127.25.94.2:56225)
0807 22:57:14.214352 (+ 3253us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:14.214379 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0807 22:57:14.214538 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:57:14.214906 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:57:14.214911 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:57:14.214925 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0807 22:57:14.215362 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:14.215374 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:14.216794 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:14.216801 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:14.218687 (+ 1886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:14.218702 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:14.219492 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:14.219501 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:14.219650 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:14.219730 (+ 80us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0807 22:57:14.220172 (+ 442us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0807 22:57:14.220185 (+ 13us) client_negotiation.cc:769] Sending connection context
0807 22:57:14.220292 (+ 107us) client_negotiation.cc:240] Negotiation successful
0807 22:57:14.220461 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3018,"thread_start_us":2863,"threads_started":1}
I20260807 22:57:14.263569 5557 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36680:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 22:57:14.263845 5557 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36680:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 22:57:14.267681 5557 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1f81ab1e409a4b718bf275a1e92f4448: Sending DeleteTablet for 3 replicas of tablet 3928efd33c5d4ca3b84760be76bdbcb8
I20260807 22:57:14.268653 6255 tablet_service.cc:1514] Processing DeleteTablet for tablet 3928efd33c5d4ca3b84760be76bdbcb8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:57:14 UTC) from {username='oryx', principal='oryx/127.25.94.62@KRBTEST.COM'} at 127.0.0.1:50446
I20260807 22:57:14.268778 6035 tablet_service.cc:1514] Processing DeleteTablet for tablet 3928efd33c5d4ca3b84760be76bdbcb8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:57:14 UTC) from {username='oryx', principal='oryx/127.25.94.62@KRBTEST.COM'} at 127.0.0.1:35548
I20260807 22:57:14.270015 5785 tablet_service.cc:1514] Processing DeleteTablet for tablet 3928efd33c5d4ca3b84760be76bdbcb8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:57:14 UTC) from {username='oryx', principal='oryx/127.25.94.62@KRBTEST.COM'} at 127.0.0.1:54806
I20260807 22:57:14.284291 6523 tablet_replica.cc:330] T 3928efd33c5d4ca3b84760be76bdbcb8 P 7a7eafb4de9948cf855043890af6dfab: stopping tablet replica
I20260807 22:57:14.284546 6523 raft_consensus.cc:2229] T 3928efd33c5d4ca3b84760be76bdbcb8 P 7a7eafb4de9948cf855043890af6dfab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 22:57:14.284826 6523 raft_consensus.cc:2258] T 3928efd33c5d4ca3b84760be76bdbcb8 P 7a7eafb4de9948cf855043890af6dfab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 22:57:14.286168 6523 ts_tablet_manager.cc:1905] T 3928efd33c5d4ca3b84760be76bdbcb8 P 7a7eafb4de9948cf855043890af6dfab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 22:57:14.293576 6523 ts_tablet_manager.cc:1918] T 3928efd33c5d4ca3b84760be76bdbcb8 P 7a7eafb4de9948cf855043890af6dfab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 22:57:14.293740 6523 log.cc:1196] T 3928efd33c5d4ca3b84760be76bdbcb8 P 7a7eafb4de9948cf855043890af6dfab: Deleting WAL directory at /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786143361779964-25976-0/minicluster-data/ts-2/wal/wals/3928efd33c5d4ca3b84760be76bdbcb8
I20260807 22:57:14.294211 6523 ts_tablet_manager.cc:1939] T 3928efd33c5d4ca3b84760be76bdbcb8 P 7a7eafb4de9948cf855043890af6dfab: Deleting consensus metadata
I20260807 22:57:14.295408 5538 catalog_manager.cc:4841] TS 7a7eafb4de9948cf855043890af6dfab (127.25.94.3:43907): tablet 3928efd33c5d4ca3b84760be76bdbcb8 (table test-table [id=2b65045699864f58b95d14cfaccf28e4]) successfully deleted
I20260807 22:57:14.300179 6525 tablet_replica.cc:330] T 3928efd33c5d4ca3b84760be76bdbcb8 P 6609f2fb140445708643ec0ce81dbdbf: stopping tablet replica
I20260807 22:57:14.300885 6525 raft_consensus.cc:2229] T 3928efd33c5d4ca3b84760be76bdbcb8 P 6609f2fb140445708643ec0ce81dbdbf [term 1 LEADER]: Raft consensus shutting down.
I20260807 22:57:14.301568 6524 tablet_replica.cc:330] T 3928efd33c5d4ca3b84760be76bdbcb8 P 29a7220b19354d949c943c99359b54ce: stopping tablet replica
I20260807 22:57:14.302218 6525 raft_consensus.cc:2258] T 3928efd33c5d4ca3b84760be76bdbcb8 P 6609f2fb140445708643ec0ce81dbdbf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 22:57:14.302539 6524 raft_consensus.cc:2229] T 3928efd33c5d4ca3b84760be76bdbcb8 P 29a7220b19354d949c943c99359b54ce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 22:57:14.302901 6524 raft_consensus.cc:2258] T 3928efd33c5d4ca3b84760be76bdbcb8 P 29a7220b19354d949c943c99359b54ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 22:57:14.304168 6525 ts_tablet_manager.cc:1905] T 3928efd33c5d4ca3b84760be76bdbcb8 P 6609f2fb140445708643ec0ce81dbdbf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 22:57:14.305526 6524 ts_tablet_manager.cc:1905] T 3928efd33c5d4ca3b84760be76bdbcb8 P 29a7220b19354d949c943c99359b54ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 22:57:14.312148 6525 ts_tablet_manager.cc:1918] T 3928efd33c5d4ca3b84760be76bdbcb8 P 6609f2fb140445708643ec0ce81dbdbf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 22:57:14.312423 6525 log.cc:1196] T 3928efd33c5d4ca3b84760be76bdbcb8 P 6609f2fb140445708643ec0ce81dbdbf: Deleting WAL directory at /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786143361779964-25976-0/minicluster-data/ts-0/wal/wals/3928efd33c5d4ca3b84760be76bdbcb8
I20260807 22:57:14.313138 6525 ts_tablet_manager.cc:1939] T 3928efd33c5d4ca3b84760be76bdbcb8 P 6609f2fb140445708643ec0ce81dbdbf: Deleting consensus metadata
I20260807 22:57:14.314621 5538 catalog_manager.cc:4841] TS 6609f2fb140445708643ec0ce81dbdbf (127.25.94.1:44883): tablet 3928efd33c5d4ca3b84760be76bdbcb8 (table test-table [id=2b65045699864f58b95d14cfaccf28e4]) successfully deleted
I20260807 22:57:14.319540 6524 ts_tablet_manager.cc:1918] T 3928efd33c5d4ca3b84760be76bdbcb8 P 29a7220b19354d949c943c99359b54ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 22:57:14.319756 6524 log.cc:1196] T 3928efd33c5d4ca3b84760be76bdbcb8 P 29a7220b19354d949c943c99359b54ce: Deleting WAL directory at /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786143361779964-25976-0/minicluster-data/ts-1/wal/wals/3928efd33c5d4ca3b84760be76bdbcb8
I20260807 22:57:14.320292 6524 ts_tablet_manager.cc:1939] T 3928efd33c5d4ca3b84760be76bdbcb8 P 29a7220b19354d949c943c99359b54ce: Deleting consensus metadata
I20260807 22:57:14.321478 5536 catalog_manager.cc:4841] TS 29a7220b19354d949c943c99359b54ce (127.25.94.2:38743): tablet 3928efd33c5d4ca3b84760be76bdbcb8 (table test-table [id=2b65045699864f58b95d14cfaccf28e4]) successfully deleted
Aug 07 22:57:14 dist-test-slave-z8x9 krb5kdc[5473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143434, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 07 22:57:14 dist-test-slave-z8x9 krb5kdc[5473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.25.94.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 22:57:14 dist-test-slave-z8x9 krb5kdc[5473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.25.94.62@KRBTEST.COM, Server not found in Kerberos database
W20260807 22:57:14.392938 6538 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:57:14.360305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36692 (local address 127.25.94.62:34919)
0807 22:57:14.377795 (+ 17490us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:14.377803 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:14.378510 (+ 707us) server_negotiation.cc:407] Connection header received
0807 22:57:14.379103 (+ 593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:14.379109 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:14.379197 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:14.379278 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:14.381361 (+ 2083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:14.382635 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:14.384586 (+ 1951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:14.385288 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:14.385386 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:14.392593 (+ 7207us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36692: BlockingRecv error: recv got EOF from 127.0.0.1:36692 (error 108)
Metrics: {"server-negotiator.queue_time_us":17322,"thread_start_us":17494,"threads_started":1}
Aug 07 22:57:14 dist-test-slave-z8x9 krb5kdc[5473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143434, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.25.94.62@KRBTEST.COM
I20260807 22:57:14.425886 6538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:14.409068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36696 (local address 127.25.94.62:34919)
0807 22:57:14.409348 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:14.409359 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:14.409887 (+ 528us) server_negotiation.cc:407] Connection header received
0807 22:57:14.410118 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:14.410122 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:14.410188 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:14.410303 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:14.411920 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:14.413013 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:14.414676 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:14.415190 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:14.415332 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:14.420006 (+ 4674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:57:14.420011 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:57:14.420014 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:57:14.420048 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:57:14.423130 (+ 3082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:14.423821 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:14.423826 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:14.423829 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:14.423953 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:14.424362 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:14.424367 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:14.424369 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:14.424651 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:57:14.424937 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0807 22:57:14.425258 (+ 321us) server_negotiation.cc:299] Negotiation successful
0807 22:57:14.425494 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260807 22:57:14.429661 5557 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36696:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 22:57:14.430344 5557 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 22:57:14.437467 5785 tablet_service.cc:1467] Processing CreateTablet for tablet 8efc82a611da4b02aee7a3bb83dd9696 (DEFAULT_TABLE table=test-table [id=7874de7432b34d06bec4973db8729119]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:57:14.437937 5785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8efc82a611da4b02aee7a3bb83dd9696. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:57:14.443470 6547 tablet_bootstrap.cc:492] T 8efc82a611da4b02aee7a3bb83dd9696 P 6609f2fb140445708643ec0ce81dbdbf: Bootstrap starting.
I20260807 22:57:14.443958 6035 tablet_service.cc:1467] Processing CreateTablet for tablet 8efc82a611da4b02aee7a3bb83dd9696 (DEFAULT_TABLE table=test-table [id=7874de7432b34d06bec4973db8729119]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:57:14.444437 6035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8efc82a611da4b02aee7a3bb83dd9696. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:57:14.445696 6547 tablet_bootstrap.cc:654] T 8efc82a611da4b02aee7a3bb83dd9696 P 6609f2fb140445708643ec0ce81dbdbf: Neither blocks nor log segments found. Creating new log.
I20260807 22:57:14.450327 6255 tablet_service.cc:1467] Processing CreateTablet for tablet 8efc82a611da4b02aee7a3bb83dd9696 (DEFAULT_TABLE table=test-table [id=7874de7432b34d06bec4973db8729119]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:57:14.450850 6255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8efc82a611da4b02aee7a3bb83dd9696. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:57:14.468120 6553 tablet_bootstrap.cc:492] T 8efc82a611da4b02aee7a3bb83dd9696 P 29a7220b19354d949c943c99359b54ce: Bootstrap starting.
I20260807 22:57:14.469620 6552 tablet_bootstrap.cc:492] T 8efc82a611da4b02aee7a3bb83dd9696 P 7a7eafb4de9948cf855043890af6dfab: Bootstrap starting.
I20260807 22:57:14.471454 6552 tablet_bootstrap.cc:654] T 8efc82a611da4b02aee7a3bb83dd9696 P 7a7eafb4de9948cf855043890af6dfab: Neither blocks nor log segments found. Creating new log.
I20260807 22:57:14.473004 6553 tablet_bootstrap.cc:654] T 8efc82a611da4b02aee7a3bb83dd9696 P 29a7220b19354d949c943c99359b54ce: Neither blocks nor log segments found. Creating new log.
I20260807 22:57:14.475083 6547 tablet_bootstrap.cc:492] T 8efc82a611da4b02aee7a3bb83dd9696 P 6609f2fb140445708643ec0ce81dbdbf: No bootstrap required, opened a new log
I20260807 22:57:14.475273 6547 ts_tablet_manager.cc:1397] T 8efc82a611da4b02aee7a3bb83dd9696 P 6609f2fb140445708643ec0ce81dbdbf: Time spent bootstrapping tablet: real 0.032s user 0.016s sys 0.012s
I20260807 22:57:14.476011 6547 raft_consensus.cc:348] T 8efc82a611da4b02aee7a3bb83dd9696 P 6609f2fb140445708643ec0ce81dbdbf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6609f2fb140445708643ec0ce81dbdbf" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44883 } } peers { permanent_uuid: "29a7220b19354d949c943c99359b54ce" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 38743 } } peers { permanent_uuid: "7a7eafb4de9948cf855043890af6dfab" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 43907 } }
I20260807 22:57:14.476277 6547 raft_consensus.cc:374] T 8efc82a611da4b02aee7a3bb83dd9696 P 6609f2fb140445708643ec0ce81dbdbf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:57:14.476375 6547 raft_consensus.cc:729] T 8efc82a611da4b02aee7a3bb83dd9696 P 6609f2fb140445708643ec0ce81dbdbf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6609f2fb140445708643ec0ce81dbdbf, State: Initialized, Role: FOLLOWER
I20260807 22:57:14.476655 6547 consensus_queue.cc:260] T 8efc82a611da4b02aee7a3bb83dd9696 P 6609f2fb140445708643ec0ce81dbdbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6609f2fb140445708643ec0ce81dbdbf" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44883 } } peers { permanent_uuid: "29a7220b19354d949c943c99359b54ce" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 38743 } } peers { permanent_uuid: "7a7eafb4de9948cf855043890af6dfab" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 43907 } }
I20260807 22:57:14.477386 6547 ts_tablet_manager.cc:1428] T 8efc82a611da4b02aee7a3bb83dd9696 P 6609f2fb140445708643ec0ce81dbdbf: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260807 22:57:14.479275 6553 tablet_bootstrap.cc:492] T 8efc82a611da4b02aee7a3bb83dd9696 P 29a7220b19354d949c943c99359b54ce: No bootstrap required, opened a new log
I20260807 22:57:14.479460 6553 ts_tablet_manager.cc:1397] T 8efc82a611da4b02aee7a3bb83dd9696 P 29a7220b19354d949c943c99359b54ce: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.000s
I20260807 22:57:14.480458 6553 raft_consensus.cc:348] T 8efc82a611da4b02aee7a3bb83dd9696 P 29a7220b19354d949c943c99359b54ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6609f2fb140445708643ec0ce81dbdbf" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44883 } } peers { permanent_uuid: "29a7220b19354d949c943c99359b54ce" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 38743 } } peers { permanent_uuid: "7a7eafb4de9948cf855043890af6dfab" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 43907 } }
I20260807 22:57:14.480772 6553 raft_consensus.cc:374] T 8efc82a611da4b02aee7a3bb83dd9696 P 29a7220b19354d949c943c99359b54ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:57:14.480901 6553 raft_consensus.cc:729] T 8efc82a611da4b02aee7a3bb83dd9696 P 29a7220b19354d949c943c99359b54ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29a7220b19354d949c943c99359b54ce, State: Initialized, Role: FOLLOWER
I20260807 22:57:14.481261 6553 consensus_queue.cc:260] T 8efc82a611da4b02aee7a3bb83dd9696 P 29a7220b19354d949c943c99359b54ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6609f2fb140445708643ec0ce81dbdbf" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44883 } } peers { permanent_uuid: "29a7220b19354d949c943c99359b54ce" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 38743 } } peers { permanent_uuid: "7a7eafb4de9948cf855043890af6dfab" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 43907 } }
I20260807 22:57:14.482844 6552 tablet_bootstrap.cc:492] T 8efc82a611da4b02aee7a3bb83dd9696 P 7a7eafb4de9948cf855043890af6dfab: No bootstrap required, opened a new log
I20260807 22:57:14.482944 6553 ts_tablet_manager.cc:1428] T 8efc82a611da4b02aee7a3bb83dd9696 P 29a7220b19354d949c943c99359b54ce: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260807 22:57:14.483012 6552 ts_tablet_manager.cc:1397] T 8efc82a611da4b02aee7a3bb83dd9696 P 7a7eafb4de9948cf855043890af6dfab: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.003s
I20260807 22:57:14.483871 6552 raft_consensus.cc:348] T 8efc82a611da4b02aee7a3bb83dd9696 P 7a7eafb4de9948cf855043890af6dfab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6609f2fb140445708643ec0ce81dbdbf" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44883 } } peers { permanent_uuid: "29a7220b19354d949c943c99359b54ce" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 38743 } } peers { permanent_uuid: "7a7eafb4de9948cf855043890af6dfab" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 43907 } }
I20260807 22:57:14.484149 6552 raft_consensus.cc:374] T 8efc82a611da4b02aee7a3bb83dd9696 P 7a7eafb4de9948cf855043890af6dfab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:57:14.484266 6552 raft_consensus.cc:729] T 8efc82a611da4b02aee7a3bb83dd9696 P 7a7eafb4de9948cf855043890af6dfab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a7eafb4de9948cf855043890af6dfab, State: Initialized, Role: FOLLOWER
I20260807 22:57:14.484601 6552 consensus_queue.cc:260] T 8efc82a611da4b02aee7a3bb83dd9696 P 7a7eafb4de9948cf855043890af6dfab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6609f2fb140445708643ec0ce81dbdbf" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44883 } } peers { permanent_uuid: "29a7220b19354d949c943c99359b54ce" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 38743 } } peers { permanent_uuid: "7a7eafb4de9948cf855043890af6dfab" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 43907 } }
I20260807 22:57:14.487717 6552 ts_tablet_manager.cc:1428] T 8efc82a611da4b02aee7a3bb83dd9696 P 7a7eafb4de9948cf855043890af6dfab: Time spent starting tablet: real 0.005s user 0.000s sys 0.003s
I20260807 22:57:14.593281 6463 raft_consensus.cc:482] T 8efc82a611da4b02aee7a3bb83dd9696 P 6609f2fb140445708643ec0ce81dbdbf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 22:57:14.593588 6463 raft_consensus.cc:504] T 8efc82a611da4b02aee7a3bb83dd9696 P 6609f2fb140445708643ec0ce81dbdbf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6609f2fb140445708643ec0ce81dbdbf" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44883 } } peers { permanent_uuid: "29a7220b19354d949c943c99359b54ce" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 38743 } } peers { permanent_uuid: "7a7eafb4de9948cf855043890af6dfab" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 43907 } }
I20260807 22:57:14.594383 6463 leader_election.cc:288] T 8efc82a611da4b02aee7a3bb83dd9696 P 6609f2fb140445708643ec0ce81dbdbf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29a7220b19354d949c943c99359b54ce (127.25.94.2:38743), 7a7eafb4de9948cf855043890af6dfab (127.25.94.3:43907)
I20260807 22:57:14.595072 6082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8efc82a611da4b02aee7a3bb83dd9696" candidate_uuid: "6609f2fb140445708643ec0ce81dbdbf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29a7220b19354d949c943c99359b54ce" is_pre_election: true
I20260807 22:57:14.595098 6292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8efc82a611da4b02aee7a3bb83dd9696" candidate_uuid: "6609f2fb140445708643ec0ce81dbdbf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a7eafb4de9948cf855043890af6dfab" is_pre_election: true
I20260807 22:57:14.595356 6082 raft_consensus.cc:2454] T 8efc82a611da4b02aee7a3bb83dd9696 P 29a7220b19354d949c943c99359b54ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6609f2fb140445708643ec0ce81dbdbf in term 0.
I20260807 22:57:14.595400 6292 raft_consensus.cc:2454] T 8efc82a611da4b02aee7a3bb83dd9696 P 7a7eafb4de9948cf855043890af6dfab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6609f2fb140445708643ec0ce81dbdbf in term 0.
I20260807 22:57:14.595866 5706 leader_election.cc:302] T 8efc82a611da4b02aee7a3bb83dd9696 P 6609f2fb140445708643ec0ce81dbdbf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29a7220b19354d949c943c99359b54ce, 6609f2fb140445708643ec0ce81dbdbf; no voters:
I20260807 22:57:14.596256 6463 raft_consensus.cc:2781] T 8efc82a611da4b02aee7a3bb83dd9696 P 6609f2fb140445708643ec0ce81dbdbf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 22:57:14.596354 6463 raft_consensus.cc:482] T 8efc82a611da4b02aee7a3bb83dd9696 P 6609f2fb140445708643ec0ce81dbdbf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 22:57:14.596483 6463 raft_consensus.cc:3037] T 8efc82a611da4b02aee7a3bb83dd9696 P 6609f2fb140445708643ec0ce81dbdbf [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:57:14.599120 6463 raft_consensus.cc:504] T 8efc82a611da4b02aee7a3bb83dd9696 P 6609f2fb140445708643ec0ce81dbdbf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6609f2fb140445708643ec0ce81dbdbf" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44883 } } peers { permanent_uuid: "29a7220b19354d949c943c99359b54ce" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 38743 } } peers { permanent_uuid: "7a7eafb4de9948cf855043890af6dfab" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 43907 } }
I20260807 22:57:14.599879 6463 leader_election.cc:288] T 8efc82a611da4b02aee7a3bb83dd9696 P 6609f2fb140445708643ec0ce81dbdbf [CANDIDATE]: Term 1 election: Requested vote from peers 29a7220b19354d949c943c99359b54ce (127.25.94.2:38743), 7a7eafb4de9948cf855043890af6dfab (127.25.94.3:43907)
I20260807 22:57:14.600454 6292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8efc82a611da4b02aee7a3bb83dd9696" candidate_uuid: "6609f2fb140445708643ec0ce81dbdbf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a7eafb4de9948cf855043890af6dfab"
I20260807 22:57:14.600471 6082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8efc82a611da4b02aee7a3bb83dd9696" candidate_uuid: "6609f2fb140445708643ec0ce81dbdbf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29a7220b19354d949c943c99359b54ce"
I20260807 22:57:14.600706 6292 raft_consensus.cc:3037] T 8efc82a611da4b02aee7a3bb83dd9696 P 7a7eafb4de9948cf855043890af6dfab [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:57:14.600718 6082 raft_consensus.cc:3037] T 8efc82a611da4b02aee7a3bb83dd9696 P 29a7220b19354d949c943c99359b54ce [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:57:14.603926 6082 raft_consensus.cc:2454] T 8efc82a611da4b02aee7a3bb83dd9696 P 29a7220b19354d949c943c99359b54ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6609f2fb140445708643ec0ce81dbdbf in term 1.
I20260807 22:57:14.603926 6292 raft_consensus.cc:2454] T 8efc82a611da4b02aee7a3bb83dd9696 P 7a7eafb4de9948cf855043890af6dfab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6609f2fb140445708643ec0ce81dbdbf in term 1.
I20260807 22:57:14.604417 5708 leader_election.cc:302] T 8efc82a611da4b02aee7a3bb83dd9696 P 6609f2fb140445708643ec0ce81dbdbf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6609f2fb140445708643ec0ce81dbdbf, 7a7eafb4de9948cf855043890af6dfab; no voters:
I20260807 22:57:14.604773 6463 raft_consensus.cc:2781] T 8efc82a611da4b02aee7a3bb83dd9696 P 6609f2fb140445708643ec0ce81dbdbf [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:57:14.604974 6463 raft_consensus.cc:686] T 8efc82a611da4b02aee7a3bb83dd9696 P 6609f2fb140445708643ec0ce81dbdbf [term 1 LEADER]: Becoming Leader. State: Replica: 6609f2fb140445708643ec0ce81dbdbf, State: Running, Role: LEADER
I20260807 22:57:14.605335 6463 consensus_queue.cc:237] T 8efc82a611da4b02aee7a3bb83dd9696 P 6609f2fb140445708643ec0ce81dbdbf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6609f2fb140445708643ec0ce81dbdbf" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44883 } } peers { permanent_uuid: "29a7220b19354d949c943c99359b54ce" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 38743 } } peers { permanent_uuid: "7a7eafb4de9948cf855043890af6dfab" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 43907 } }
I20260807 22:57:14.608492 5560 catalog_manager.cc:5495] T 8efc82a611da4b02aee7a3bb83dd9696 P 6609f2fb140445708643ec0ce81dbdbf reported cstate change: term changed from 0 to 1, leader changed from <none> to 6609f2fb140445708643ec0ce81dbdbf (127.25.94.1). New cstate: current_term: 1 leader_uuid: "6609f2fb140445708643ec0ce81dbdbf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6609f2fb140445708643ec0ce81dbdbf" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44883 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "29a7220b19354d949c943c99359b54ce" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 38743 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a7eafb4de9948cf855043890af6dfab" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 43907 } health_report { overall_health: UNKNOWN } } }
I20260807 22:57:14.623903 6508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:14.618481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54822 (local address 127.25.94.1:44883)
0807 22:57:14.618736 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:14.618745 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:14.618779 (+ 34us) server_negotiation.cc:407] Connection header received
0807 22:57:14.619009 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:14.619014 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:14.619086 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:14.619200 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 22:57:14.619595 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:14.620386 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:14.621904 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:14.622419 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:14.622570 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:14.623076 (+ 506us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 22:57:14.623218 (+ 142us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 22:57:14.623325 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0807 22:57:14.623456 (+ 131us) server_negotiation.cc:299] Negotiation successful
0807 22:57:14.623542 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260807 22:57:14.626996 6292 raft_consensus.cc:1261] T 8efc82a611da4b02aee7a3bb83dd9696 P 7a7eafb4de9948cf855043890af6dfab [term 1 FOLLOWER]: Refusing update from remote peer 6609f2fb140445708643ec0ce81dbdbf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 22:57:14.627066 6082 raft_consensus.cc:1261] T 8efc82a611da4b02aee7a3bb83dd9696 P 29a7220b19354d949c943c99359b54ce [term 1 FOLLOWER]: Refusing update from remote peer 6609f2fb140445708643ec0ce81dbdbf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 22:57:14.627761 6463 consensus_queue.cc:1035] T 8efc82a611da4b02aee7a3bb83dd9696 P 6609f2fb140445708643ec0ce81dbdbf [LEADER]: Connected to new peer: Peer: permanent_uuid: "29a7220b19354d949c943c99359b54ce" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 38743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 22:57:14.628074 6517 consensus_queue.cc:1035] T 8efc82a611da4b02aee7a3bb83dd9696 P 6609f2fb140445708643ec0ce81dbdbf [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a7eafb4de9948cf855043890af6dfab" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 43907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 22:57:14.651211 5557 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36696:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 22:57:14.651513 5557 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36696:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 22:57:14.655369 5557 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1f81ab1e409a4b718bf275a1e92f4448: Sending DeleteTablet for 3 replicas of tablet 8efc82a611da4b02aee7a3bb83dd9696
I20260807 22:57:14.656433 5785 tablet_service.cc:1514] Processing DeleteTablet for tablet 8efc82a611da4b02aee7a3bb83dd9696 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:57:14 UTC) from {username='oryx', principal='oryx/127.25.94.62@KRBTEST.COM'} at 127.0.0.1:54806
I20260807 22:57:14.656654 6035 tablet_service.cc:1514] Processing DeleteTablet for tablet 8efc82a611da4b02aee7a3bb83dd9696 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:57:14 UTC) from {username='oryx', principal='oryx/127.25.94.62@KRBTEST.COM'} at 127.0.0.1:35548
I20260807 22:57:14.656744 6525 tablet_replica.cc:330] T 8efc82a611da4b02aee7a3bb83dd9696 P 6609f2fb140445708643ec0ce81dbdbf: stopping tablet replica
I20260807 22:57:14.657021 6524 tablet_replica.cc:330] T 8efc82a611da4b02aee7a3bb83dd9696 P 29a7220b19354d949c943c99359b54ce: stopping tablet replica
I20260807 22:57:14.657280 6524 raft_consensus.cc:2229] T 8efc82a611da4b02aee7a3bb83dd9696 P 29a7220b19354d949c943c99359b54ce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 22:57:14.657552 6524 raft_consensus.cc:2258] T 8efc82a611da4b02aee7a3bb83dd9696 P 29a7220b19354d949c943c99359b54ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 22:57:14.657862 6255 tablet_service.cc:1514] Processing DeleteTablet for tablet 8efc82a611da4b02aee7a3bb83dd9696 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:57:14 UTC) from {username='oryx', principal='oryx/127.25.94.62@KRBTEST.COM'} at 127.0.0.1:50446
I20260807 22:57:14.658051 6035 tablet_service.cc:1514] Processing DeleteTablet for tablet 8efc82a611da4b02aee7a3bb83dd9696 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:57:14 UTC) from {username='oryx', principal='oryx/127.25.94.62@KRBTEST.COM'} at 127.0.0.1:35548
I20260807 22:57:14.658368 6523 tablet_replica.cc:330] T 8efc82a611da4b02aee7a3bb83dd9696 P 7a7eafb4de9948cf855043890af6dfab: stopping tablet replica
I20260807 22:57:14.658584 6523 raft_consensus.cc:2229] T 8efc82a611da4b02aee7a3bb83dd9696 P 7a7eafb4de9948cf855043890af6dfab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 22:57:14.658809 6523 raft_consensus.cc:2258] T 8efc82a611da4b02aee7a3bb83dd9696 P 7a7eafb4de9948cf855043890af6dfab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 22:57:14.659422 6525 raft_consensus.cc:2229] T 8efc82a611da4b02aee7a3bb83dd9696 P 6609f2fb140445708643ec0ce81dbdbf [term 1 LEADER]: Raft consensus shutting down.
I20260807 22:57:14.659775 6525 raft_consensus.cc:2258] T 8efc82a611da4b02aee7a3bb83dd9696 P 6609f2fb140445708643ec0ce81dbdbf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 22:57:14.661530 6523 ts_tablet_manager.cc:1905] T 8efc82a611da4b02aee7a3bb83dd9696 P 7a7eafb4de9948cf855043890af6dfab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 22:57:14.665833 6524 ts_tablet_manager.cc:1905] T 8efc82a611da4b02aee7a3bb83dd9696 P 29a7220b19354d949c943c99359b54ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 22:57:14.670023 6525 ts_tablet_manager.cc:1905] T 8efc82a611da4b02aee7a3bb83dd9696 P 6609f2fb140445708643ec0ce81dbdbf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 22:57:14.670827 5557 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36696:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 22:57:14.671627 5557 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 22:57:14.677323 6523 ts_tablet_manager.cc:1918] T 8efc82a611da4b02aee7a3bb83dd9696 P 7a7eafb4de9948cf855043890af6dfab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 22:57:14.677579 6523 log.cc:1196] T 8efc82a611da4b02aee7a3bb83dd9696 P 7a7eafb4de9948cf855043890af6dfab: Deleting WAL directory at /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786143361779964-25976-0/minicluster-data/ts-2/wal/wals/8efc82a611da4b02aee7a3bb83dd9696
I20260807 22:57:14.678165 6523 ts_tablet_manager.cc:1939] T 8efc82a611da4b02aee7a3bb83dd9696 P 7a7eafb4de9948cf855043890af6dfab: Deleting consensus metadata
I20260807 22:57:14.679057 6525 ts_tablet_manager.cc:1918] T 8efc82a611da4b02aee7a3bb83dd9696 P 6609f2fb140445708643ec0ce81dbdbf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 22:57:14.679697 5538 catalog_manager.cc:4841] TS 7a7eafb4de9948cf855043890af6dfab (127.25.94.3:43907): tablet 8efc82a611da4b02aee7a3bb83dd9696 (table test-table [id=7874de7432b34d06bec4973db8729119]) successfully deleted
I20260807 22:57:14.681265 6035 tablet_service.cc:1467] Processing CreateTablet for tablet 87e7078c68c24471a1d6f98ef5e4d1fd (DEFAULT_TABLE table=test-table [id=0cdffad4ff0d480fb0682268add1dcc0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:57:14.681594 5785 tablet_service.cc:1467] Processing CreateTablet for tablet 87e7078c68c24471a1d6f98ef5e4d1fd (DEFAULT_TABLE table=test-table [id=0cdffad4ff0d480fb0682268add1dcc0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:57:14.681754 6035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87e7078c68c24471a1d6f98ef5e4d1fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:57:14.682016 5785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87e7078c68c24471a1d6f98ef5e4d1fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:57:14.682540 6524 ts_tablet_manager.cc:1918] T 8efc82a611da4b02aee7a3bb83dd9696 P 29a7220b19354d949c943c99359b54ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 22:57:14.682749 6524 log.cc:1196] T 8efc82a611da4b02aee7a3bb83dd9696 P 29a7220b19354d949c943c99359b54ce: Deleting WAL directory at /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786143361779964-25976-0/minicluster-data/ts-1/wal/wals/8efc82a611da4b02aee7a3bb83dd9696
I20260807 22:57:14.683250 6524 ts_tablet_manager.cc:1939] T 8efc82a611da4b02aee7a3bb83dd9696 P 29a7220b19354d949c943c99359b54ce: Deleting consensus metadata
I20260807 22:57:14.684746 5536 catalog_manager.cc:4841] TS 29a7220b19354d949c943c99359b54ce (127.25.94.2:38743): tablet 8efc82a611da4b02aee7a3bb83dd9696 (table test-table [id=7874de7432b34d06bec4973db8729119]) successfully deleted
I20260807 22:57:14.684999 6525 log.cc:1196] T 8efc82a611da4b02aee7a3bb83dd9696 P 6609f2fb140445708643ec0ce81dbdbf: Deleting WAL directory at /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786143361779964-25976-0/minicluster-data/ts-0/wal/wals/8efc82a611da4b02aee7a3bb83dd9696
I20260807 22:57:14.685539 6525 ts_tablet_manager.cc:1939] T 8efc82a611da4b02aee7a3bb83dd9696 P 6609f2fb140445708643ec0ce81dbdbf: Deleting consensus metadata
I20260807 22:57:14.685881 6255 tablet_service.cc:1467] Processing CreateTablet for tablet 87e7078c68c24471a1d6f98ef5e4d1fd (DEFAULT_TABLE table=test-table [id=0cdffad4ff0d480fb0682268add1dcc0]), partition=RANGE (key) PARTITION UNBOUNDED
W20260807 22:57:14.685935 5536 catalog_manager.cc:4808] TS 29a7220b19354d949c943c99359b54ce (127.25.94.2:38743): delete failed for tablet 8efc82a611da4b02aee7a3bb83dd9696 because the tablet was not found. No further retry: Not found: Tablet not found: 8efc82a611da4b02aee7a3bb83dd9696
I20260807 22:57:14.686321 6255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87e7078c68c24471a1d6f98ef5e4d1fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:57:14.687106 5538 catalog_manager.cc:4841] TS 6609f2fb140445708643ec0ce81dbdbf (127.25.94.1:44883): tablet 8efc82a611da4b02aee7a3bb83dd9696 (table test-table [id=7874de7432b34d06bec4973db8729119]) successfully deleted
I20260807 22:57:14.688764 6547 tablet_bootstrap.cc:492] T 87e7078c68c24471a1d6f98ef5e4d1fd P 6609f2fb140445708643ec0ce81dbdbf: Bootstrap starting.
I20260807 22:57:14.690626 6547 tablet_bootstrap.cc:654] T 87e7078c68c24471a1d6f98ef5e4d1fd P 6609f2fb140445708643ec0ce81dbdbf: Neither blocks nor log segments found. Creating new log.
I20260807 22:57:14.691807 6553 tablet_bootstrap.cc:492] T 87e7078c68c24471a1d6f98ef5e4d1fd P 29a7220b19354d949c943c99359b54ce: Bootstrap starting.
I20260807 22:57:14.693539 6553 tablet_bootstrap.cc:654] T 87e7078c68c24471a1d6f98ef5e4d1fd P 29a7220b19354d949c943c99359b54ce: Neither blocks nor log segments found. Creating new log.
I20260807 22:57:14.693980 6552 tablet_bootstrap.cc:492] T 87e7078c68c24471a1d6f98ef5e4d1fd P 7a7eafb4de9948cf855043890af6dfab: Bootstrap starting.
I20260807 22:57:14.695737 6553 tablet_bootstrap.cc:492] T 87e7078c68c24471a1d6f98ef5e4d1fd P 29a7220b19354d949c943c99359b54ce: No bootstrap required, opened a new log
I20260807 22:57:14.695780 6552 tablet_bootstrap.cc:654] T 87e7078c68c24471a1d6f98ef5e4d1fd P 7a7eafb4de9948cf855043890af6dfab: Neither blocks nor log segments found. Creating new log.
I20260807 22:57:14.695796 6547 tablet_bootstrap.cc:492] T 87e7078c68c24471a1d6f98ef5e4d1fd P 6609f2fb140445708643ec0ce81dbdbf: No bootstrap required, opened a new log
I20260807 22:57:14.695916 6553 ts_tablet_manager.cc:1397] T 87e7078c68c24471a1d6f98ef5e4d1fd P 29a7220b19354d949c943c99359b54ce: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260807 22:57:14.695973 6547 ts_tablet_manager.cc:1397] T 87e7078c68c24471a1d6f98ef5e4d1fd P 6609f2fb140445708643ec0ce81dbdbf: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20260807 22:57:14.696741 6553 raft_consensus.cc:348] T 87e7078c68c24471a1d6f98ef5e4d1fd P 29a7220b19354d949c943c99359b54ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29a7220b19354d949c943c99359b54ce" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 38743 } } peers { permanent_uuid: "6609f2fb140445708643ec0ce81dbdbf" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44883 } } peers { permanent_uuid: "7a7eafb4de9948cf855043890af6dfab" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 43907 } }
I20260807 22:57:14.696825 6547 raft_consensus.cc:348] T 87e7078c68c24471a1d6f98ef5e4d1fd P 6609f2fb140445708643ec0ce81dbdbf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29a7220b19354d949c943c99359b54ce" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 38743 } } peers { permanent_uuid: "6609f2fb140445708643ec0ce81dbdbf" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44883 } } peers { permanent_uuid: "7a7eafb4de9948cf855043890af6dfab" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 43907 } }
I20260807 22:57:14.697018 6553 raft_consensus.cc:374] T 87e7078c68c24471a1d6f98ef5e4d1fd P 29a7220b19354d949c943c99359b54ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:57:14.697124 6547 raft_consensus.cc:374] T 87e7078c68c24471a1d6f98ef5e4d1fd P 6609f2fb140445708643ec0ce81dbdbf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:57:14.697116 6553 raft_consensus.cc:729] T 87e7078c68c24471a1d6f98ef5e4d1fd P 29a7220b19354d949c943c99359b54ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29a7220b19354d949c943c99359b54ce, State: Initialized, Role: FOLLOWER
I20260807 22:57:14.697279 6547 raft_consensus.cc:729] T 87e7078c68c24471a1d6f98ef5e4d1fd P 6609f2fb140445708643ec0ce81dbdbf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6609f2fb140445708643ec0ce81dbdbf, State: Initialized, Role: FOLLOWER
I20260807 22:57:14.697486 6553 consensus_queue.cc:260] T 87e7078c68c24471a1d6f98ef5e4d1fd P 29a7220b19354d949c943c99359b54ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29a7220b19354d949c943c99359b54ce" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 38743 } } peers { permanent_uuid: "6609f2fb140445708643ec0ce81dbdbf" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44883 } } peers { permanent_uuid: "7a7eafb4de9948cf855043890af6dfab" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 43907 } }
I20260807 22:57:14.697597 6547 consensus_queue.cc:260] T 87e7078c68c24471a1d6f98ef5e4d1fd P 6609f2fb140445708643ec0ce81dbdbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29a7220b19354d949c943c99359b54ce" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 38743 } } peers { permanent_uuid: "6609f2fb140445708643ec0ce81dbdbf" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44883 } } peers { permanent_uuid: "7a7eafb4de9948cf855043890af6dfab" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 43907 } }
I20260807 22:57:14.699828 6553 ts_tablet_manager.cc:1428] T 87e7078c68c24471a1d6f98ef5e4d1fd P 29a7220b19354d949c943c99359b54ce: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260807 22:57:14.700757 6547 ts_tablet_manager.cc:1428] T 87e7078c68c24471a1d6f98ef5e4d1fd P 6609f2fb140445708643ec0ce81dbdbf: Time spent starting tablet: real 0.005s user 0.001s sys 0.002s
I20260807 22:57:14.705968 6552 tablet_bootstrap.cc:492] T 87e7078c68c24471a1d6f98ef5e4d1fd P 7a7eafb4de9948cf855043890af6dfab: No bootstrap required, opened a new log
I20260807 22:57:14.706146 6552 ts_tablet_manager.cc:1397] T 87e7078c68c24471a1d6f98ef5e4d1fd P 7a7eafb4de9948cf855043890af6dfab: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.009s
I20260807 22:57:14.706787 6552 raft_consensus.cc:348] T 87e7078c68c24471a1d6f98ef5e4d1fd P 7a7eafb4de9948cf855043890af6dfab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29a7220b19354d949c943c99359b54ce" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 38743 } } peers { permanent_uuid: "6609f2fb140445708643ec0ce81dbdbf" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44883 } } peers { permanent_uuid: "7a7eafb4de9948cf855043890af6dfab" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 43907 } }
I20260807 22:57:14.706979 6552 raft_consensus.cc:374] T 87e7078c68c24471a1d6f98ef5e4d1fd P 7a7eafb4de9948cf855043890af6dfab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:57:14.707074 6552 raft_consensus.cc:729] T 87e7078c68c24471a1d6f98ef5e4d1fd P 7a7eafb4de9948cf855043890af6dfab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a7eafb4de9948cf855043890af6dfab, State: Initialized, Role: FOLLOWER
I20260807 22:57:14.707355 6552 consensus_queue.cc:260] T 87e7078c68c24471a1d6f98ef5e4d1fd P 7a7eafb4de9948cf855043890af6dfab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29a7220b19354d949c943c99359b54ce" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 38743 } } peers { permanent_uuid: "6609f2fb140445708643ec0ce81dbdbf" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44883 } } peers { permanent_uuid: "7a7eafb4de9948cf855043890af6dfab" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 43907 } }
I20260807 22:57:14.707875 6552 ts_tablet_manager.cc:1428] T 87e7078c68c24471a1d6f98ef5e4d1fd P 7a7eafb4de9948cf855043890af6dfab: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260807 22:57:14.747648 6463 raft_consensus.cc:482] T 87e7078c68c24471a1d6f98ef5e4d1fd P 6609f2fb140445708643ec0ce81dbdbf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 22:57:14.747920 6463 raft_consensus.cc:504] T 87e7078c68c24471a1d6f98ef5e4d1fd P 6609f2fb140445708643ec0ce81dbdbf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29a7220b19354d949c943c99359b54ce" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 38743 } } peers { permanent_uuid: "6609f2fb140445708643ec0ce81dbdbf" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44883 } } peers { permanent_uuid: "7a7eafb4de9948cf855043890af6dfab" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 43907 } }
I20260807 22:57:14.748754 6463 leader_election.cc:288] T 87e7078c68c24471a1d6f98ef5e4d1fd P 6609f2fb140445708643ec0ce81dbdbf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29a7220b19354d949c943c99359b54ce (127.25.94.2:38743), 7a7eafb4de9948cf855043890af6dfab (127.25.94.3:43907)
I20260807 22:57:14.749363 6082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87e7078c68c24471a1d6f98ef5e4d1fd" candidate_uuid: "6609f2fb140445708643ec0ce81dbdbf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29a7220b19354d949c943c99359b54ce" is_pre_election: true
I20260807 22:57:14.749491 6292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87e7078c68c24471a1d6f98ef5e4d1fd" candidate_uuid: "6609f2fb140445708643ec0ce81dbdbf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a7eafb4de9948cf855043890af6dfab" is_pre_election: true
I20260807 22:57:14.749665 6082 raft_consensus.cc:2454] T 87e7078c68c24471a1d6f98ef5e4d1fd P 29a7220b19354d949c943c99359b54ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6609f2fb140445708643ec0ce81dbdbf in term 0.
I20260807 22:57:14.749725 6292 raft_consensus.cc:2454] T 87e7078c68c24471a1d6f98ef5e4d1fd P 7a7eafb4de9948cf855043890af6dfab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6609f2fb140445708643ec0ce81dbdbf in term 0.
I20260807 22:57:14.750349 5708 leader_election.cc:302] T 87e7078c68c24471a1d6f98ef5e4d1fd P 6609f2fb140445708643ec0ce81dbdbf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 29a7220b19354d949c943c99359b54ce, 6609f2fb140445708643ec0ce81dbdbf, 7a7eafb4de9948cf855043890af6dfab; no voters:
I20260807 22:57:14.750730 6463 raft_consensus.cc:2781] T 87e7078c68c24471a1d6f98ef5e4d1fd P 6609f2fb140445708643ec0ce81dbdbf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 22:57:14.750865 6463 raft_consensus.cc:482] T 87e7078c68c24471a1d6f98ef5e4d1fd P 6609f2fb140445708643ec0ce81dbdbf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 22:57:14.751008 6463 raft_consensus.cc:3037] T 87e7078c68c24471a1d6f98ef5e4d1fd P 6609f2fb140445708643ec0ce81dbdbf [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:57:14.754333 6463 raft_consensus.cc:504] T 87e7078c68c24471a1d6f98ef5e4d1fd P 6609f2fb140445708643ec0ce81dbdbf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29a7220b19354d949c943c99359b54ce" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 38743 } } peers { permanent_uuid: "6609f2fb140445708643ec0ce81dbdbf" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44883 } } peers { permanent_uuid: "7a7eafb4de9948cf855043890af6dfab" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 43907 } }
I20260807 22:57:14.755061 6463 leader_election.cc:288] T 87e7078c68c24471a1d6f98ef5e4d1fd P 6609f2fb140445708643ec0ce81dbdbf [CANDIDATE]: Term 1 election: Requested vote from peers 29a7220b19354d949c943c99359b54ce (127.25.94.2:38743), 7a7eafb4de9948cf855043890af6dfab (127.25.94.3:43907)
I20260807 22:57:14.755553 6082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87e7078c68c24471a1d6f98ef5e4d1fd" candidate_uuid: "6609f2fb140445708643ec0ce81dbdbf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29a7220b19354d949c943c99359b54ce"
I20260807 22:57:14.755609 6292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87e7078c68c24471a1d6f98ef5e4d1fd" candidate_uuid: "6609f2fb140445708643ec0ce81dbdbf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a7eafb4de9948cf855043890af6dfab"
I20260807 22:57:14.755770 6082 raft_consensus.cc:3037] T 87e7078c68c24471a1d6f98ef5e4d1fd P 29a7220b19354d949c943c99359b54ce [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:57:14.755837 6292 raft_consensus.cc:3037] T 87e7078c68c24471a1d6f98ef5e4d1fd P 7a7eafb4de9948cf855043890af6dfab [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:57:14.757927 6082 raft_consensus.cc:2454] T 87e7078c68c24471a1d6f98ef5e4d1fd P 29a7220b19354d949c943c99359b54ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6609f2fb140445708643ec0ce81dbdbf in term 1.
I20260807 22:57:14.758445 5706 leader_election.cc:302] T 87e7078c68c24471a1d6f98ef5e4d1fd P 6609f2fb140445708643ec0ce81dbdbf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29a7220b19354d949c943c99359b54ce, 6609f2fb140445708643ec0ce81dbdbf; no voters:
I20260807 22:57:14.758746 6463 raft_consensus.cc:2781] T 87e7078c68c24471a1d6f98ef5e4d1fd P 6609f2fb140445708643ec0ce81dbdbf [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:57:14.758947 6463 raft_consensus.cc:686] T 87e7078c68c24471a1d6f98ef5e4d1fd P 6609f2fb140445708643ec0ce81dbdbf [term 1 LEADER]: Becoming Leader. State: Replica: 6609f2fb140445708643ec0ce81dbdbf, State: Running, Role: LEADER
I20260807 22:57:14.759016 6292 raft_consensus.cc:2454] T 87e7078c68c24471a1d6f98ef5e4d1fd P 7a7eafb4de9948cf855043890af6dfab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6609f2fb140445708643ec0ce81dbdbf in term 1.
I20260807 22:57:14.759205 6463 consensus_queue.cc:237] T 87e7078c68c24471a1d6f98ef5e4d1fd P 6609f2fb140445708643ec0ce81dbdbf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29a7220b19354d949c943c99359b54ce" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 38743 } } peers { permanent_uuid: "6609f2fb140445708643ec0ce81dbdbf" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44883 } } peers { permanent_uuid: "7a7eafb4de9948cf855043890af6dfab" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 43907 } }
I20260807 22:57:14.762480 5557 catalog_manager.cc:5495] T 87e7078c68c24471a1d6f98ef5e4d1fd P 6609f2fb140445708643ec0ce81dbdbf reported cstate change: term changed from 0 to 1, leader changed from <none> to 6609f2fb140445708643ec0ce81dbdbf (127.25.94.1). New cstate: current_term: 1 leader_uuid: "6609f2fb140445708643ec0ce81dbdbf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29a7220b19354d949c943c99359b54ce" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 38743 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6609f2fb140445708643ec0ce81dbdbf" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44883 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a7eafb4de9948cf855043890af6dfab" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 43907 } health_report { overall_health: UNKNOWN } } }
I20260807 22:57:14.801510 6292 raft_consensus.cc:1261] T 87e7078c68c24471a1d6f98ef5e4d1fd P 7a7eafb4de9948cf855043890af6dfab [term 1 FOLLOWER]: Refusing update from remote peer 6609f2fb140445708643ec0ce81dbdbf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 22:57:14.801551 6082 raft_consensus.cc:1261] T 87e7078c68c24471a1d6f98ef5e4d1fd P 29a7220b19354d949c943c99359b54ce [term 1 FOLLOWER]: Refusing update from remote peer 6609f2fb140445708643ec0ce81dbdbf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 22:57:14.802086 6463 consensus_queue.cc:1035] T 87e7078c68c24471a1d6f98ef5e4d1fd P 6609f2fb140445708643ec0ce81dbdbf [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a7eafb4de9948cf855043890af6dfab" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 43907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 22:57:14.802465 6429 consensus_queue.cc:1035] T 87e7078c68c24471a1d6f98ef5e4d1fd P 6609f2fb140445708643ec0ce81dbdbf [LEADER]: Connected to new peer: Peer: permanent_uuid: "29a7220b19354d949c943c99359b54ce" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 38743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 22:57:14.864360 5557 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36696:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 22:57:14.864591 5557 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36696:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 22:57:14.867543 5557 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1f81ab1e409a4b718bf275a1e92f4448: Sending DeleteTablet for 3 replicas of tablet 87e7078c68c24471a1d6f98ef5e4d1fd
I20260807 22:57:14.868405 5785 tablet_service.cc:1514] Processing DeleteTablet for tablet 87e7078c68c24471a1d6f98ef5e4d1fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:57:14 UTC) from {username='oryx', principal='oryx/127.25.94.62@KRBTEST.COM'} at 127.0.0.1:54806
I20260807 22:57:14.868415 6035 tablet_service.cc:1514] Processing DeleteTablet for tablet 87e7078c68c24471a1d6f98ef5e4d1fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:57:14 UTC) from {username='oryx', principal='oryx/127.25.94.62@KRBTEST.COM'} at 127.0.0.1:35548
I20260807 22:57:14.868724 6525 tablet_replica.cc:330] T 87e7078c68c24471a1d6f98ef5e4d1fd P 6609f2fb140445708643ec0ce81dbdbf: stopping tablet replica
I20260807 22:57:14.868746 6255 tablet_service.cc:1514] Processing DeleteTablet for tablet 87e7078c68c24471a1d6f98ef5e4d1fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:57:14 UTC) from {username='oryx', principal='oryx/127.25.94.62@KRBTEST.COM'} at 127.0.0.1:50446
I20260807 22:57:14.868932 6525 raft_consensus.cc:2229] T 87e7078c68c24471a1d6f98ef5e4d1fd P 6609f2fb140445708643ec0ce81dbdbf [term 1 LEADER]: Raft consensus shutting down.
I20260807 22:57:14.869077 6524 tablet_replica.cc:330] T 87e7078c68c24471a1d6f98ef5e4d1fd P 29a7220b19354d949c943c99359b54ce: stopping tablet replica
I20260807 22:57:14.869078 6523 tablet_replica.cc:330] T 87e7078c68c24471a1d6f98ef5e4d1fd P 7a7eafb4de9948cf855043890af6dfab: stopping tablet replica
I20260807 22:57:14.869283 6524 raft_consensus.cc:2229] T 87e7078c68c24471a1d6f98ef5e4d1fd P 29a7220b19354d949c943c99359b54ce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 22:57:14.869323 6525 raft_consensus.cc:2258] T 87e7078c68c24471a1d6f98ef5e4d1fd P 6609f2fb140445708643ec0ce81dbdbf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 22:57:14.869333 6523 raft_consensus.cc:2229] T 87e7078c68c24471a1d6f98ef5e4d1fd P 7a7eafb4de9948cf855043890af6dfab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 22:57:14.869513 6524 raft_consensus.cc:2258] T 87e7078c68c24471a1d6f98ef5e4d1fd P 29a7220b19354d949c943c99359b54ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 22:57:14.869557 6523 raft_consensus.cc:2258] T 87e7078c68c24471a1d6f98ef5e4d1fd P 7a7eafb4de9948cf855043890af6dfab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 22:57:14.870362 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 5678
I20260807 22:57:14.870776 6525 ts_tablet_manager.cc:1905] T 87e7078c68c24471a1d6f98ef5e4d1fd P 6609f2fb140445708643ec0ce81dbdbf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 22:57:14.870965 6524 ts_tablet_manager.cc:1905] T 87e7078c68c24471a1d6f98ef5e4d1fd P 29a7220b19354d949c943c99359b54ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 22:57:14.871665 6523 ts_tablet_manager.cc:1905] T 87e7078c68c24471a1d6f98ef5e4d1fd P 7a7eafb4de9948cf855043890af6dfab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 22:57:14.878847 6524 ts_tablet_manager.cc:1918] T 87e7078c68c24471a1d6f98ef5e4d1fd P 29a7220b19354d949c943c99359b54ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 22:57:14.878968 6523 ts_tablet_manager.cc:1918] T 87e7078c68c24471a1d6f98ef5e4d1fd P 7a7eafb4de9948cf855043890af6dfab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 22:57:14.879096 6524 log.cc:1196] T 87e7078c68c24471a1d6f98ef5e4d1fd P 29a7220b19354d949c943c99359b54ce: Deleting WAL directory at /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786143361779964-25976-0/minicluster-data/ts-1/wal/wals/87e7078c68c24471a1d6f98ef5e4d1fd
I20260807 22:57:14.879230 6523 log.cc:1196] T 87e7078c68c24471a1d6f98ef5e4d1fd P 7a7eafb4de9948cf855043890af6dfab: Deleting WAL directory at /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786143361779964-25976-0/minicluster-data/ts-2/wal/wals/87e7078c68c24471a1d6f98ef5e4d1fd
I20260807 22:57:14.879640 6524 ts_tablet_manager.cc:1939] T 87e7078c68c24471a1d6f98ef5e4d1fd P 29a7220b19354d949c943c99359b54ce: Deleting consensus metadata
I20260807 22:57:14.879814 6523 ts_tablet_manager.cc:1939] T 87e7078c68c24471a1d6f98ef5e4d1fd P 7a7eafb4de9948cf855043890af6dfab: Deleting consensus metadata
I20260807 22:57:14.879807 6525 ts_tablet_manager.cc:1918] T 87e7078c68c24471a1d6f98ef5e4d1fd P 6609f2fb140445708643ec0ce81dbdbf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 22:57:14.879978 6525 log.cc:1196] T 87e7078c68c24471a1d6f98ef5e4d1fd P 6609f2fb140445708643ec0ce81dbdbf: Deleting WAL directory at /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786143361779964-25976-0/minicluster-data/ts-0/wal/wals/87e7078c68c24471a1d6f98ef5e4d1fd
I20260807 22:57:14.880518 6525 ts_tablet_manager.cc:1939] T 87e7078c68c24471a1d6f98ef5e4d1fd P 6609f2fb140445708643ec0ce81dbdbf: Deleting consensus metadata
I20260807 22:57:14.880961 5536 catalog_manager.cc:4841] TS 29a7220b19354d949c943c99359b54ce (127.25.94.2:38743): tablet 87e7078c68c24471a1d6f98ef5e4d1fd (table test-table [id=0cdffad4ff0d480fb0682268add1dcc0]) successfully deleted
I20260807 22:57:14.881130 5538 catalog_manager.cc:4841] TS 7a7eafb4de9948cf855043890af6dfab (127.25.94.3:43907): tablet 87e7078c68c24471a1d6f98ef5e4d1fd (table test-table [id=0cdffad4ff0d480fb0682268add1dcc0]) successfully deleted
I20260807 22:57:14.881579 5538 catalog_manager.cc:4841] TS 6609f2fb140445708643ec0ce81dbdbf (127.25.94.1:44883): tablet 87e7078c68c24471a1d6f98ef5e4d1fd (table test-table [id=0cdffad4ff0d480fb0682268add1dcc0]) successfully deleted
Aug 07 22:57:14 dist-test-slave-z8x9 krb5kdc[5473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143434, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.25.94.1@KRBTEST.COM
I20260807 22:57:14.895422 6508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:14.873793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54838 (local address 127.25.94.1:44883)
0807 22:57:14.873984 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:14.873990 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:14.881235 (+ 7245us) server_negotiation.cc:407] Connection header received
0807 22:57:14.881531 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:14.881537 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:14.881621 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:14.881741 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:14.883002 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:14.883718 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:14.884978 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:14.885479 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:14.885619 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:14.889729 (+ 4110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:57:14.889745 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:57:14.889762 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:57:14.889795 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:57:14.892560 (+ 2765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:14.893270 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:14.893280 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:14.893284 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:14.893418 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:14.893816 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:14.893821 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:14.893828 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:14.894198 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:57:14.894509 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0807 22:57:14.894843 (+ 334us) server_negotiation.cc:299] Negotiation successful
0807 22:57:14.895086 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260807 22:57:14.896075 5904 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 22:57:15.022589 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 5678
I20260807 22:57:15.054628 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 5918
Aug 07 22:57:15 dist-test-slave-z8x9 krb5kdc[5473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143434, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.25.94.2@KRBTEST.COM
I20260807 22:57:15.070291 6588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:15.055426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35570 (local address 127.25.94.2:38743)
0807 22:57:15.055971 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:15.055976 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:15.055993 (+ 17us) server_negotiation.cc:407] Connection header received
0807 22:57:15.056049 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:15.056053 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:15.056119 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:15.056212 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:15.057518 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:15.058292 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:15.059517 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:15.059925 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:15.060023 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:15.064493 (+ 4470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:57:15.064509 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:57:15.064525 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:57:15.064562 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:57:15.067610 (+ 3048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:15.068265 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:15.068273 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:15.068275 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:15.068429 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:15.068772 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:15.068775 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:15.068777 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:15.069127 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:57:15.069472 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0807 22:57:15.069818 (+ 346us) server_negotiation.cc:299] Negotiation successful
0807 22:57:15.070004 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":304,"threads_started":1}
I20260807 22:57:15.071019 6141 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 22:57:15.222252 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 5918
I20260807 22:57:15.257997 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 6154
Aug 07 22:57:15 dist-test-slave-z8x9 krb5kdc[5473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143434, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.25.94.3@KRBTEST.COM
I20260807 22:57:15.273943 6591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:15.258781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50456 (local address 127.25.94.3:43907)
0807 22:57:15.259308 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:15.259313 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:15.259328 (+ 15us) server_negotiation.cc:407] Connection header received
0807 22:57:15.259389 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:15.259393 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:15.259454 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:15.259585 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:15.260927 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:15.261740 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:15.262983 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:15.263393 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:15.263538 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:15.268064 (+ 4526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:57:15.268079 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:57:15.268095 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:57:15.268131 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:57:15.270993 (+ 2862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:15.271676 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:15.271684 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:15.271687 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:15.271830 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:15.272218 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:15.272224 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:15.272229 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:15.272608 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:57:15.272930 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0807 22:57:15.273269 (+ 339us) server_negotiation.cc:299] Negotiation successful
0807 22:57:15.273526 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":241,"threads_started":1}
I20260807 22:57:15.274695 6359 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 22:57:15.373430 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 6154
I20260807 22:57:15.402372 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 5508
I20260807 22:57:15.403782 5618 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 22:57:15.526356 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 5508
2026-08-07T22:57:15Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (6250 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786143361779964-25976-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 22:57:15 dist-test-slave-z8x9 krb5kdc[6600](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 22:57:15 dist-test-slave-z8x9 krb5kdc[6600](info): set up 2 sockets
Aug 07 22:57:15 dist-test-slave-z8x9 krb5kdc[6600](info): commencing operation
krb5kdc: starting...
W20260807 22:57:16.903379 25976 mini_kdc.cc:121] Time spent starting KDC: real 1.323s user 0.007s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786143361779964-25976-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786143361779964-25976-0/krb5kdc/test-user.keytab.
Aug 07 22:57:17 dist-test-slave-z8x9 krb5kdc[6600](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143437, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T22:57:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T22:57:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.94.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.62.keytab.
Entry for principal kudu/127.25.94.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.62.keytab.
I20260807 22:57:17.177561 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786143361779964-25976-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786143361779964-25976-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786143361779964-25976-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.62:44965
--webserver_interface=127.25.94.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.62.keytab
--principal=kudu/127.25.94.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.20:41697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.94.62:44965
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786143361779964-25976-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786143361779964-25976-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786143361779964-25976-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:57:17.526234 6626 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:57:17.526538 6626 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:57:17.526695 6626 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:57:17.533924 6626 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 22:57:17.534039 6626 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:57:17.534132 6626 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:57:17.534279 6626 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 22:57:17.534358 6626 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 22:57:17.544411 6626 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:41697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786143361779964-25976-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786143361779964-25976-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.94.62:44965
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.62:44965
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.94.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786143361779964-25976-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 22:57:17.545773 6626 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:57:17.547165 6626 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:57:17.559042 6639 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:57:17.559096 6640 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:57:17.559130 6643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:57:17.560024 6626 server_base.cc:1034] running on GCE node
I20260807 22:57:17.560882 6626 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:57:17.562346 6626 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:57:17.563566 6626 hybrid_clock.cc:648] HybridClock initialized: now 1786143437563533 us; error 45 us; skew 500 ppm
I20260807 22:57:17.563876 6626 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-07T22:57:17Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (2104 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 22:57:17 dist-test-slave-z8x9 krb5kdc[6653](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 22:57:17 dist-test-slave-z8x9 krb5kdc[6653](info): set up 2 sockets
Aug 07 22:57:17 dist-test-slave-z8x9 krb5kdc[6653](info): commencing operation
krb5kdc: starting...
W20260807 22:57:18.985479 25976 mini_kdc.cc:121] Time spent starting KDC: real 1.301s user 0.002s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/krb5kdc/test-user.keytab.
Aug 07 22:57:19 dist-test-slave-z8x9 krb5kdc[6653](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143439, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T22:57:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T22:57:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.94.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.62.keytab.
Entry for principal kudu/127.25.94.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.62.keytab.
I20260807 22:57:19.266618 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.62:44503
--webserver_interface=127.25.94.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.62.keytab
--principal=kudu/127.25.94.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.20:38859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.94.62:44503
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:57:19.610711 6675 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:57:19.611029 6675 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:57:19.611187 6675 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:57:19.618528 6675 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 22:57:19.618656 6675 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:57:19.618779 6675 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:57:19.618882 6675 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 22:57:19.618958 6675 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 22:57:19.629245 6675 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:38859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.94.62:44503
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.62.keytab
--principal=kudu/127.25.94.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.62:44503
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.94.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 22:57:19.630676 6675 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:57:19.632089 6675 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:57:19.645399 6688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:57:19.645490 6681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:57:19.645455 6691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:57:19.646238 6675 server_base.cc:1034] running on GCE node
I20260807 22:57:19.647014 6675 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:57:19.648411 6675 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:57:19.649622 6675 hybrid_clock.cc:648] HybridClock initialized: now 1786143439649583 us; error 28 us; skew 500 ppm
I20260807 22:57:19.649978 6675 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 22:57:19 dist-test-slave-z8x9 krb5kdc[6653](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143439, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:57:19.653131 6675 init.cc:377] Logged in from keytab as kudu/127.25.94.62@KRBTEST.COM (short username kudu)
I20260807 22:57:19.654680 6675 webserver.cc:460] Webserver started at http://127.25.94.62:36365/ using document root <none> and password file <none>
I20260807 22:57:19.655385 6675 fs_manager.cc:359] Metadata directory not provided
I20260807 22:57:19.655516 6675 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:57:19.655862 6675 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:57:19.659032 6675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/minicluster-data/master-0/data/instance:
uuid: "27f5d6942f0544838999b7466df2b0aa"
format_stamp: "Formatted at 2026-08-07 22:57:19 on dist-test-slave-z8x9"
server_key: "8002477171e63b29a3d12aea1b195d6e"
server_key_iv: "2c4d0bd8b888ce9ce2f10a61e6d4a9b1"
server_key_version: "encryptionkey@0"
I20260807 22:57:19.659888 6675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/minicluster-data/master-0/wal/instance:
uuid: "27f5d6942f0544838999b7466df2b0aa"
format_stamp: "Formatted at 2026-08-07 22:57:19 on dist-test-slave-z8x9"
server_key: "8002477171e63b29a3d12aea1b195d6e"
server_key_iv: "2c4d0bd8b888ce9ce2f10a61e6d4a9b1"
server_key_version: "encryptionkey@0"
I20260807 22:57:19.665445 6675 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260807 22:57:19.669723 6699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:57:19.671099 6675 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260807 22:57:19.671283 6675 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/minicluster-data/master-0/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/minicluster-data/master-0/wal
uuid: "27f5d6942f0544838999b7466df2b0aa"
format_stamp: "Formatted at 2026-08-07 22:57:19 on dist-test-slave-z8x9"
server_key: "8002477171e63b29a3d12aea1b195d6e"
server_key_iv: "2c4d0bd8b888ce9ce2f10a61e6d4a9b1"
server_key_version: "encryptionkey@0"
I20260807 22:57:19.671495 6675 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:57:19.689786 6675 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:57:19.694207 6675 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:57:19.694509 6675 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:57:19.714004 6675 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.62:44503
I20260807 22:57:19.714043 6802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.62:44503 every 8 connection(s)
I20260807 22:57:19.715456 6675 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
I20260807 22:57:19.719547 6805 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:57:19.723728 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 6675
I20260807 22:57:19.723973 25976 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/minicluster-data/master-0/wal/instance
I20260807 22:57:19.724798 25976 external_mini_cluster.cc:1428] Setting key aa286d5b5bcc110389fb00c031337744
I20260807 22:57:19.734175 6805 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27f5d6942f0544838999b7466df2b0aa: Bootstrap starting.
I20260807 22:57:19.738158 6805 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 27f5d6942f0544838999b7466df2b0aa: Neither blocks nor log segments found. Creating new log.
I20260807 22:57:19.739241 6805 log.cc:824] T 00000000000000000000000000000000 P 27f5d6942f0544838999b7466df2b0aa: Log is configured to *not* fsync() on all Append() calls
Aug 07 22:57:19 dist-test-slave-z8x9 krb5kdc[6653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143439, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.62@KRBTEST.COM
I20260807 22:57:19.742627 6805 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27f5d6942f0544838999b7466df2b0aa: No bootstrap required, opened a new log
I20260807 22:57:19.749217 6808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:19.726987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50314 (local address 127.25.94.62:44503)
0807 22:57:19.727782 (+ 795us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:19.727793 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:19.727841 (+ 48us) server_negotiation.cc:407] Connection header received
0807 22:57:19.729610 (+ 1769us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:19.729634 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:19.730042 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:19.730373 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:19.732304 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:19.733977 (+ 1673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:19.736002 (+ 2025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:19.737091 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:19.737338 (+ 247us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:19.741784 (+ 4446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:57:19.741793 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:57:19.741814 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:57:19.741850 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:57:19.744980 (+ 3130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:19.745876 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:19.745882 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:19.745889 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:19.746057 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:19.746481 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:19.746484 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:19.746486 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:19.747005 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:57:19.747224 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0807 22:57:19.747728 (+ 504us) server_negotiation.cc:299] Negotiation successful
0807 22:57:19.748103 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":589,"thread_start_us":385,"threads_started":1}
I20260807 22:57:19.751755 6805 raft_consensus.cc:348] T 00000000000000000000000000000000 P 27f5d6942f0544838999b7466df2b0aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27f5d6942f0544838999b7466df2b0aa" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 44503 } }
I20260807 22:57:19.752139 6805 raft_consensus.cc:374] T 00000000000000000000000000000000 P 27f5d6942f0544838999b7466df2b0aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:57:19.752266 6805 raft_consensus.cc:729] T 00000000000000000000000000000000 P 27f5d6942f0544838999b7466df2b0aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27f5d6942f0544838999b7466df2b0aa, State: Initialized, Role: FOLLOWER
I20260807 22:57:19.752883 6805 consensus_queue.cc:260] T 00000000000000000000000000000000 P 27f5d6942f0544838999b7466df2b0aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27f5d6942f0544838999b7466df2b0aa" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 44503 } }
I20260807 22:57:19.753140 6805 raft_consensus.cc:388] T 00000000000000000000000000000000 P 27f5d6942f0544838999b7466df2b0aa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 22:57:19.753314 6805 raft_consensus.cc:482] T 00000000000000000000000000000000 P 27f5d6942f0544838999b7466df2b0aa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 22:57:19.753495 6805 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 27f5d6942f0544838999b7466df2b0aa [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:57:19.755667 6805 raft_consensus.cc:504] T 00000000000000000000000000000000 P 27f5d6942f0544838999b7466df2b0aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27f5d6942f0544838999b7466df2b0aa" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 44503 } }
I20260807 22:57:19.756184 6805 leader_election.cc:302] T 00000000000000000000000000000000 P 27f5d6942f0544838999b7466df2b0aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 27f5d6942f0544838999b7466df2b0aa; no voters:
I20260807 22:57:19.756821 6805 leader_election.cc:288] T 00000000000000000000000000000000 P 27f5d6942f0544838999b7466df2b0aa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 22:57:19.757091 6811 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 27f5d6942f0544838999b7466df2b0aa [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:57:19.758100 6811 raft_consensus.cc:686] T 00000000000000000000000000000000 P 27f5d6942f0544838999b7466df2b0aa [term 1 LEADER]: Becoming Leader. State: Replica: 27f5d6942f0544838999b7466df2b0aa, State: Running, Role: LEADER
I20260807 22:57:19.758720 6811 consensus_queue.cc:237] T 00000000000000000000000000000000 P 27f5d6942f0544838999b7466df2b0aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27f5d6942f0544838999b7466df2b0aa" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 44503 } }
I20260807 22:57:19.759136 6805 sys_catalog.cc:564] T 00000000000000000000000000000000 P 27f5d6942f0544838999b7466df2b0aa [sys.catalog]: configured and running, proceeding with master startup.
I20260807 22:57:19.761548 6813 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27f5d6942f0544838999b7466df2b0aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "27f5d6942f0544838999b7466df2b0aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27f5d6942f0544838999b7466df2b0aa" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 44503 } } }
I20260807 22:57:19.761982 6813 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27f5d6942f0544838999b7466df2b0aa [sys.catalog]: This master's current role is: LEADER
I20260807 22:57:19.761902 6814 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27f5d6942f0544838999b7466df2b0aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 27f5d6942f0544838999b7466df2b0aa. Latest consensus state: current_term: 1 leader_uuid: "27f5d6942f0544838999b7466df2b0aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27f5d6942f0544838999b7466df2b0aa" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 44503 } } }
I20260807 22:57:19.762163 6814 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27f5d6942f0544838999b7466df2b0aa [sys.catalog]: This master's current role is: LEADER
I20260807 22:57:19.765053 6820 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 22:57:19.769502 6820 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 22:57:19.778684 6820 catalog_manager.cc:1347] Generated new cluster ID: 8028894b86714d738551c09cae84a81d
I20260807 22:57:19.778865 6820 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 22:57:19.798933 6820 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 22:57:19.800429 6820 catalog_manager.cc:1504] Loading token signing keys...
I20260807 22:57:19.811151 6820 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 27f5d6942f0544838999b7466df2b0aa: Generated new TSK 0
I20260807 22:57:19.812186 6820 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.94.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.1.keytab.
Entry for principal kudu/127.25.94.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.1.keytab.
I20260807 22:57:19.881541 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.1:0
--local_ip_for_outbound_sockets=127.25.94.1
--webserver_interface=127.25.94.1
--webserver_port=0
--tserver_master_addrs=127.25.94.62:44503
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.1.keytab
--principal=kudu/127.25.94.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.20:38859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:57:20.244252 6846 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:57:20.244741 6846 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:57:20.244907 6846 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:57:20.252815 6846 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:57:20.252981 6846 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:57:20.253105 6846 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.1
I20260807 22:57:20.263855 6846 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:38859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.1.keytab
--principal=kudu/127.25.94.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.94.1
--webserver_port=0
--tserver_master_addrs=127.25.94.62:44503
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.1
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 22:57:20.265430 6846 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:57:20.266907 6846 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:57:20.282410 6853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:57:20.282454 6858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:57:20.282501 6854 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:57:20.283490 6846 server_base.cc:1034] running on GCE node
I20260807 22:57:20.284214 6846 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:57:20.285372 6846 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:57:20.286640 6846 hybrid_clock.cc:648] HybridClock initialized: now 1786143440286568 us; error 65 us; skew 500 ppm
I20260807 22:57:20.287015 6846 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 22:57:20 dist-test-slave-z8x9 krb5kdc[6653](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143440, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:57:20.290761 6846 init.cc:377] Logged in from keytab as kudu/127.25.94.1@KRBTEST.COM (short username kudu)
I20260807 22:57:20.292488 6846 webserver.cc:460] Webserver started at http://127.25.94.1:46779/ using document root <none> and password file <none>
I20260807 22:57:20.293327 6846 fs_manager.cc:359] Metadata directory not provided
I20260807 22:57:20.293486 6846 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:57:20.293838 6846 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:57:20.296948 6846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/minicluster-data/ts-0/data/instance:
uuid: "060a4bf42b3d4eb3a8855c021faee130"
format_stamp: "Formatted at 2026-08-07 22:57:20 on dist-test-slave-z8x9"
server_key: "cafe2402d1022765e3694f1c8c7b8c09"
server_key_iv: "5b6873e15de3f1641ddbc8cf801a90e6"
server_key_version: "encryptionkey@0"
I20260807 22:57:20.297886 6846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/minicluster-data/ts-0/wal/instance:
uuid: "060a4bf42b3d4eb3a8855c021faee130"
format_stamp: "Formatted at 2026-08-07 22:57:20 on dist-test-slave-z8x9"
server_key: "cafe2402d1022765e3694f1c8c7b8c09"
server_key_iv: "5b6873e15de3f1641ddbc8cf801a90e6"
server_key_version: "encryptionkey@0"
I20260807 22:57:20.304303 6846 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260807 22:57:20.308240 6872 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:57:20.309948 6846 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 22:57:20.310158 6846 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/minicluster-data/ts-0/wal
uuid: "060a4bf42b3d4eb3a8855c021faee130"
format_stamp: "Formatted at 2026-08-07 22:57:20 on dist-test-slave-z8x9"
server_key: "cafe2402d1022765e3694f1c8c7b8c09"
server_key_iv: "5b6873e15de3f1641ddbc8cf801a90e6"
server_key_version: "encryptionkey@0"
I20260807 22:57:20.310387 6846 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:57:20.344686 6846 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:57:20.349000 6846 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:57:20.349360 6846 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:57:20.350338 6846 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:57:20.351758 6846 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:57:20.351867 6846 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:57:20.352026 6846 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:57:20.352101 6846 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:57:20.386844 6846 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.1:39569
I20260807 22:57:20.386940 7051 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.1:39569 every 8 connection(s)
I20260807 22:57:20.388154 6846 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/minicluster-data/ts-0/data/info.pb
I20260807 22:57:20.389034 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 6846
I20260807 22:57:20.389371 25976 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/minicluster-data/ts-0/wal/instance
I20260807 22:57:20.390084 25976 external_mini_cluster.cc:1428] Setting key e0d40e28fb280d4fc9436536a651a623
Aug 07 22:57:20 dist-test-slave-z8x9 krb5kdc[6653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143440, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.1@KRBTEST.COM for kudu/127.25.94.62@KRBTEST.COM
I20260807 22:57:20.413939 7056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:20.390902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.1:48043 (local address 127.25.94.62:44503)
0807 22:57:20.391415 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:20.391420 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:20.392671 (+ 1251us) server_negotiation.cc:407] Connection header received
0807 22:57:20.395683 (+ 3012us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:20.395687 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:20.395753 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:20.395853 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:20.398134 (+ 2281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:20.398880 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:20.401411 (+ 2531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:20.401822 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:20.401924 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:20.406011 (+ 4087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:57:20.406020 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:57:20.406028 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:57:20.406073 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:57:20.409798 (+ 3725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:20.410702 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:20.410709 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:20.410713 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:20.410906 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:20.411491 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:20.411496 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:20.411499 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:20.411892 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:57:20.412070 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0807 22:57:20.413133 (+ 1063us) server_negotiation.cc:299] Negotiation successful
0807 22:57:20.413446 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":291,"threads_started":1}
I20260807 22:57:20.415212 7057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:20.391591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:44503 (local address 127.25.94.1:48043)
0807 22:57:20.392484 (+ 893us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:20.392542 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0807 22:57:20.395369 (+ 2827us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:57:20.396141 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:57:20.396155 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:57:20.397004 (+ 849us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:57:20.397933 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:20.397960 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:20.399073 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:20.399081 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:20.401260 (+ 2179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:20.401279 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:20.402072 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:20.402082 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:20.402416 (+ 334us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:20.403189 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:57:20.403218 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:57:20.405744 (+ 2526us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:57:20.409954 (+ 4210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:57:20.409962 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:57:20.409975 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:57:20.410527 (+ 552us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:57:20.411020 (+ 493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:57:20.411028 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:57:20.411034 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:57:20.411294 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:57:20.412045 (+ 751us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:57:20.412061 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:57:20.412611 (+ 550us) client_negotiation.cc:769] Sending connection context
0807 22:57:20.412933 (+ 322us) client_negotiation.cc:240] Negotiation successful
0807 22:57:20.413580 (+ 647us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":607,"thread_start_us":408,"threads_started":1}
I20260807 22:57:20.417444 7052 heartbeater.cc:347] Connected to a master server at 127.25.94.62:44503
I20260807 22:57:20.417810 7052 heartbeater.cc:464] Registering TS with master...
I20260807 22:57:20.418787 7052 heartbeater.cc:511] Master 127.25.94.62:44503 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.25.94.2@KRBTEST.COM; defaulting to no policy
I20260807 22:57:20.421465 6731 ts_manager.cc:194] Registered new tserver with Master: 060a4bf42b3d4eb3a8855c021faee130 (127.25.94.1:39569)
I20260807 22:57:20.423115 6731 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.94.1@KRBTEST.COM'} at 127.25.94.1:48043
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.2.keytab.
Entry for principal kudu/127.25.94.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.2.keytab.
I20260807 22:57:20.456851 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.2:0
--local_ip_for_outbound_sockets=127.25.94.2
--webserver_interface=127.25.94.2
--webserver_port=0
--tserver_master_addrs=127.25.94.62:44503
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.2.keytab
--principal=kudu/127.25.94.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.20:38859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:57:20.803191 7060 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:57:20.803546 7060 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:57:20.803727 7060 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:57:20.811219 7060 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:57:20.811388 7060 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:57:20.811563 7060 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.2
I20260807 22:57:20.821676 7060 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:38859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.2.keytab
--principal=kudu/127.25.94.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.94.2
--webserver_port=0
--tserver_master_addrs=127.25.94.62:44503
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.2
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 22:57:20.823093 7060 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:57:20.824494 7060 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:57:20.838445 7070 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:57:20.838497 7066 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:57:20.838475 7077 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:57:20.839336 7060 server_base.cc:1034] running on GCE node
I20260807 22:57:20.839973 7060 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:57:20.841020 7060 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:57:20.842224 7060 hybrid_clock.cc:648] HybridClock initialized: now 1786143440842198 us; error 65 us; skew 500 ppm
I20260807 22:57:20.842561 7060 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 22:57:20 dist-test-slave-z8x9 krb5kdc[6653](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143440, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:57:20.845924 7060 init.cc:377] Logged in from keytab as kudu/127.25.94.2@KRBTEST.COM (short username kudu)
I20260807 22:57:20.847344 7060 webserver.cc:460] Webserver started at http://127.25.94.2:37755/ using document root <none> and password file <none>
I20260807 22:57:20.848048 7060 fs_manager.cc:359] Metadata directory not provided
I20260807 22:57:20.848205 7060 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:57:20.848520 7060 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:57:20.851593 7060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/minicluster-data/ts-1/data/instance:
uuid: "6c98b9c6759544f2bb23b4534d83c837"
format_stamp: "Formatted at 2026-08-07 22:57:20 on dist-test-slave-z8x9"
server_key: "15fa569ffd76bd432f418e9376591901"
server_key_iv: "b81ac75a99d87569d8193f7fe5d5efbc"
server_key_version: "encryptionkey@0"
I20260807 22:57:20.852401 7060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/minicluster-data/ts-1/wal/instance:
uuid: "6c98b9c6759544f2bb23b4534d83c837"
format_stamp: "Formatted at 2026-08-07 22:57:20 on dist-test-slave-z8x9"
server_key: "15fa569ffd76bd432f418e9376591901"
server_key_iv: "b81ac75a99d87569d8193f7fe5d5efbc"
server_key_version: "encryptionkey@0"
I20260807 22:57:20.857584 7060 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260807 22:57:20.861357 7091 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:57:20.862828 7060 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 22:57:20.863008 7060 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/minicluster-data/ts-1/wal
uuid: "6c98b9c6759544f2bb23b4534d83c837"
format_stamp: "Formatted at 2026-08-07 22:57:20 on dist-test-slave-z8x9"
server_key: "15fa569ffd76bd432f418e9376591901"
server_key_iv: "b81ac75a99d87569d8193f7fe5d5efbc"
server_key_version: "encryptionkey@0"
I20260807 22:57:20.863255 7060 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:57:20.895172 7060 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:57:20.900137 7060 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:57:20.900484 7060 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:57:20.901566 7060 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:57:20.903086 7060 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:57:20.903178 7060 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:57:20.903362 7060 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:57:20.903424 7060 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:57:20.938607 7060 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.2:39931
I20260807 22:57:20.938637 7288 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.2:39931 every 8 connection(s)
I20260807 22:57:20.939936 7060 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/minicluster-data/ts-1/data/info.pb
I20260807 22:57:20.942992 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 7060
I20260807 22:57:20.943176 25976 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/minicluster-data/ts-1/wal/instance
I20260807 22:57:20.943900 25976 external_mini_cluster.cc:1428] Setting key 3fd07cb5d75c9769056ba4b95c73332b
Aug 07 22:57:20 dist-test-slave-z8x9 krb5kdc[6653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143440, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.2@KRBTEST.COM for kudu/127.25.94.62@KRBTEST.COM
I20260807 22:57:20.962651 7296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:20.942500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.2:58599 (local address 127.25.94.62:44503)
0807 22:57:20.942967 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:20.942972 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:20.945192 (+ 2220us) server_negotiation.cc:407] Connection header received
0807 22:57:20.947305 (+ 2113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:20.947312 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:20.947403 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:20.947533 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:20.949026 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:20.949871 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:20.951417 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:20.951857 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:20.951999 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:20.955867 (+ 3868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:57:20.955872 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:57:20.955875 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:57:20.955910 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:57:20.959085 (+ 3175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:20.960082 (+ 997us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:20.960092 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:20.960097 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:20.960243 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:20.960721 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:20.960729 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:20.960733 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:20.961019 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:57:20.961185 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0807 22:57:20.961977 (+ 792us) server_negotiation.cc:299] Negotiation successful
0807 22:57:20.962243 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":274,"threads_started":1}
I20260807 22:57:20.963836 7297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:20.943049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:44503 (local address 127.25.94.2:58599)
0807 22:57:20.944974 (+ 1925us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:20.945035 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0807 22:57:20.947026 (+ 1991us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:57:20.947770 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:57:20.947778 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:57:20.948282 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:57:20.948835 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:20.948853 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:20.950028 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:20.950034 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:20.951261 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:20.951274 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:20.952098 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:20.952106 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:20.952386 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:20.953102 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:57:20.953124 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:57:20.955616 (+ 2492us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:57:20.959380 (+ 3764us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:57:20.959394 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:57:20.959410 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:57:20.959940 (+ 530us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:57:20.960374 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:57:20.960381 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:57:20.960385 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:57:20.960598 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:57:20.961157 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:57:20.961165 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:57:20.961587 (+ 422us) client_negotiation.cc:769] Sending connection context
0807 22:57:20.961877 (+ 290us) client_negotiation.cc:240] Negotiation successful
0807 22:57:20.962463 (+ 586us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1596,"thread_start_us":1361,"threads_started":1}
I20260807 22:57:20.965538 7291 heartbeater.cc:347] Connected to a master server at 127.25.94.62:44503
I20260807 22:57:20.965927 7291 heartbeater.cc:464] Registering TS with master...
I20260807 22:57:20.967027 7291 heartbeater.cc:511] Master 127.25.94.62:44503 requested a full tablet report, sending...
I20260807 22:57:20.968825 6731 ts_manager.cc:194] Registered new tserver with Master: 6c98b9c6759544f2bb23b4534d83c837 (127.25.94.2:39931)
I20260807 22:57:20.969743 6731 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.94.2@KRBTEST.COM'} at 127.25.94.2:58599
WARNING: no policy specified for kudu/127.25.94.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.3.keytab.
Entry for principal kudu/127.25.94.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.3.keytab.
I20260807 22:57:21.005517 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.3:0
--local_ip_for_outbound_sockets=127.25.94.3
--webserver_interface=127.25.94.3
--webserver_port=0
--tserver_master_addrs=127.25.94.62:44503
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.3.keytab
--principal=kudu/127.25.94.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.20:38859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:57:21.364372 7302 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:57:21.364650 7302 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:57:21.364745 7302 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:57:21.372354 7302 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:57:21.372469 7302 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:57:21.372581 7302 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.3
I20260807 22:57:21.382663 7302 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:38859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.3.keytab
--principal=kudu/127.25.94.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.94.3
--webserver_port=0
--tserver_master_addrs=127.25.94.62:44503
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.3
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 22:57:21.383912 7302 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:57:21.385232 7302 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:57:21.399108 7309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:57:21.399194 7313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:57:21.399180 7311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:57:21.402635 7302 server_base.cc:1034] running on GCE node
I20260807 22:57:21.403369 7302 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:57:21.404417 7302 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:57:21.405647 7302 hybrid_clock.cc:648] HybridClock initialized: now 1786143441405586 us; error 62 us; skew 500 ppm
I20260807 22:57:21.405974 7302 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 22:57:21 dist-test-slave-z8x9 krb5kdc[6653](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143441, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:57:21.409407 7302 init.cc:377] Logged in from keytab as kudu/127.25.94.3@KRBTEST.COM (short username kudu)
I20260807 22:57:21.410982 7302 webserver.cc:460] Webserver started at http://127.25.94.3:35477/ using document root <none> and password file <none>
I20260807 22:57:21.411672 7302 fs_manager.cc:359] Metadata directory not provided
I20260807 22:57:21.411770 7302 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:57:21.412039 7302 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:57:21.415104 7302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/minicluster-data/ts-2/data/instance:
uuid: "613b6c8ff6d54551931c538ce7d9d9b1"
format_stamp: "Formatted at 2026-08-07 22:57:21 on dist-test-slave-z8x9"
server_key: "47e27998c80c1bcb028f34bdf792e764"
server_key_iv: "25483f3b839cd93e42f5a84cb5d5b3f7"
server_key_version: "encryptionkey@0"
I20260807 22:57:21.415922 7302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/minicluster-data/ts-2/wal/instance:
uuid: "613b6c8ff6d54551931c538ce7d9d9b1"
format_stamp: "Formatted at 2026-08-07 22:57:21 on dist-test-slave-z8x9"
server_key: "47e27998c80c1bcb028f34bdf792e764"
server_key_iv: "25483f3b839cd93e42f5a84cb5d5b3f7"
server_key_version: "encryptionkey@0"
I20260807 22:57:21.421345 7302 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260807 22:57:21.424693 7325 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:57:21.426079 7302 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 22:57:21.426218 7302 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/minicluster-data/ts-2/wal
uuid: "613b6c8ff6d54551931c538ce7d9d9b1"
format_stamp: "Formatted at 2026-08-07 22:57:21 on dist-test-slave-z8x9"
server_key: "47e27998c80c1bcb028f34bdf792e764"
server_key_iv: "25483f3b839cd93e42f5a84cb5d5b3f7"
server_key_version: "encryptionkey@0"
I20260807 22:57:21.426447 7302 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:57:21.426715 7052 heartbeater.cc:503] Master 127.25.94.62:44503 was elected leader, sending a full tablet report...
I20260807 22:57:21.448558 7302 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:57:21.453227 7302 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:57:21.453511 7302 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:57:21.454485 7302 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:57:21.455905 7302 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:57:21.455979 7302 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:57:21.456066 7302 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:57:21.456110 7302 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:57:21.490114 7302 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.3:44957
I20260807 22:57:21.490218 7496 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.3:44957 every 8 connection(s)
I20260807 22:57:21.491442 7302 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/minicluster-data/ts-2/data/info.pb
I20260807 22:57:21.491780 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 7302
I20260807 22:57:21.491966 25976 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/minicluster-data/ts-2/wal/instance
I20260807 22:57:21.492615 25976 external_mini_cluster.cc:1428] Setting key 6dc853b2e22631e128a51e97ddb8cd4e
Aug 07 22:57:21 dist-test-slave-z8x9 krb5kdc[6653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143441, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.3@KRBTEST.COM for kudu/127.25.94.62@KRBTEST.COM
I20260807 22:57:21.512626 7501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:21.494313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.3:46065 (local address 127.25.94.62:44503)
0807 22:57:21.494819 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:21.494826 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:21.495841 (+ 1015us) server_negotiation.cc:407] Connection header received
0807 22:57:21.497937 (+ 2096us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:21.497941 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:21.498025 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:21.498129 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:21.499621 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:21.500462 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:21.502077 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:21.502506 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:21.502635 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:21.506343 (+ 3708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:57:21.506348 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:57:21.506352 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:57:21.506383 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:57:21.509880 (+ 3497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:21.510575 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:21.510580 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:21.510582 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:21.510705 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:21.511115 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:21.511119 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:21.511122 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:21.511411 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:57:21.511550 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0807 22:57:21.512148 (+ 598us) server_negotiation.cc:299] Negotiation successful
0807 22:57:21.512343 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":546,"threads_started":1}
I20260807 22:57:21.513633 7506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:21.494794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:44503 (local address 127.25.94.3:46065)
0807 22:57:21.495660 (+ 866us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:21.495719 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0807 22:57:21.497648 (+ 1929us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:57:21.498352 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:57:21.498360 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:57:21.498880 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:57:21.499435 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:21.499453 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:21.500594 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:21.500598 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:21.501928 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:21.501941 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:21.502777 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:21.502783 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:21.502981 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:21.503642 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:57:21.503667 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:57:21.506121 (+ 2454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:57:21.510003 (+ 3882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:57:21.510010 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:57:21.510022 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:57:21.510433 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:57:21.510800 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:57:21.510804 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:57:21.510807 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:57:21.510983 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:57:21.511529 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:57:21.511535 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:57:21.511830 (+ 295us) client_negotiation.cc:769] Sending connection context
0807 22:57:21.512049 (+ 219us) client_negotiation.cc:240] Negotiation successful
0807 22:57:21.512419 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":360,"threads_started":1}
I20260807 22:57:21.515072 7499 heartbeater.cc:347] Connected to a master server at 127.25.94.62:44503
I20260807 22:57:21.515421 7499 heartbeater.cc:464] Registering TS with master...
I20260807 22:57:21.516187 7499 heartbeater.cc:511] Master 127.25.94.62:44503 requested a full tablet report, sending...
I20260807 22:57:21.517690 6731 ts_manager.cc:194] Registered new tserver with Master: 613b6c8ff6d54551931c538ce7d9d9b1 (127.25.94.3:44957)
I20260807 22:57:21.518486 6731 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.94.3@KRBTEST.COM'} at 127.25.94.3:46065
I20260807 22:57:21.521615 25976 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.25.94.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.25.94.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.25.94.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/krb5kdc/oryx_127.25.94.1.keytab.
Entry for principal oryx/127.25.94.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/krb5kdc/oryx_127.25.94.1.keytab.
WARNING: no policy specified for oryx/127.25.94.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.25.94.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.25.94.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/krb5kdc/oryx_127.25.94.2.keytab.
Entry for principal oryx/127.25.94.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/krb5kdc/oryx_127.25.94.2.keytab.
WARNING: no policy specified for oryx/127.25.94.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.25.94.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.25.94.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/krb5kdc/oryx_127.25.94.3.keytab.
Entry for principal oryx/127.25.94.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/krb5kdc/oryx_127.25.94.3.keytab.
I20260807 22:57:21.689141 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 6846
Aug 07 22:57:21 dist-test-slave-z8x9 krb5kdc[6653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143439, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.1@KRBTEST.COM
I20260807 22:57:21.705346 7517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:21.690419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50292 (local address 127.25.94.1:39569)
0807 22:57:21.691077 (+ 658us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:21.691084 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:21.691114 (+ 30us) server_negotiation.cc:407] Connection header received
0807 22:57:21.691178 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:21.691184 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:21.691416 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:21.691572 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:21.693026 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:21.694125 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:21.695424 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:21.695982 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:21.696084 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:21.700128 (+ 4044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:57:21.700139 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:57:21.700153 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:57:21.700185 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:57:21.703007 (+ 2822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:21.703605 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:21.703612 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:21.703614 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:21.703732 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:21.704033 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:21.704036 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:21.704038 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:21.704348 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:57:21.704570 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0807 22:57:21.704768 (+ 198us) server_negotiation.cc:299] Negotiation successful
0807 22:57:21.704958 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":506,"thread_start_us":336,"threads_started":1}
I20260807 22:57:21.706588 7046 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 22:57:21.790639 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 6846
I20260807 22:57:21.814814 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 7060
Aug 07 22:57:21 dist-test-slave-z8x9 krb5kdc[6653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143439, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.2@KRBTEST.COM
I20260807 22:57:21.831422 7519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:21.815782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45800 (local address 127.25.94.2:39931)
0807 22:57:21.816292 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:21.816308 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:21.816332 (+ 24us) server_negotiation.cc:407] Connection header received
0807 22:57:21.816396 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:21.816402 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:21.816605 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:21.816786 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:21.818128 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:21.819129 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:21.820423 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:21.820965 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:21.821105 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:21.825428 (+ 4323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:57:21.825440 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:57:21.825454 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:57:21.825489 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:57:21.828315 (+ 2826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:21.828997 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:21.829004 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:21.829007 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:21.829130 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:21.829644 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:21.829650 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:21.829654 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:21.829995 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:57:21.830382 (+ 387us) server_negotiation.cc:1035] Waiting for connection context
0807 22:57:21.830705 (+ 323us) server_negotiation.cc:299] Negotiation successful
0807 22:57:21.830974 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":248,"threads_started":1}
I20260807 22:57:21.832719 7282 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 22:57:21.915759 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 7060
I20260807 22:57:21.939927 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 7302
Aug 07 22:57:21 dist-test-slave-z8x9 krb5kdc[6653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143439, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.3@KRBTEST.COM
I20260807 22:57:21.957790 7521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:21.940930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38016 (local address 127.25.94.3:44957)
0807 22:57:21.941509 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:21.941517 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:21.941550 (+ 33us) server_negotiation.cc:407] Connection header received
0807 22:57:21.941613 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:21.941620 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:21.941854 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:21.941985 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:21.943249 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:21.944247 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:21.945681 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:21.946239 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:21.946341 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:21.951768 (+ 5427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:57:21.951785 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:57:21.951802 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:57:21.951842 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:57:21.954776 (+ 2934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:21.955448 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:21.955456 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:21.955458 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:21.955583 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:21.955980 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:21.955985 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:21.955987 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:21.956406 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:57:21.956663 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0807 22:57:21.957011 (+ 348us) server_negotiation.cc:299] Negotiation successful
0807 22:57:21.957334 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":423,"thread_start_us":275,"threads_started":1}
I20260807 22:57:21.959121 7490 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 22:57:22.042626 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 7302
I20260807 22:57:22.065594 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 6675
I20260807 22:57:22.066955 6790 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 22:57:22.172888 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 6675
I20260807 22:57:22.206787 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.62:44503
--webserver_interface=127.25.94.62
--webserver_port=36365
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.62.keytab
--principal=kudu/127.25.94.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.20:38859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.94.62:44503
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:57:22.556028 7534 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:57:22.556403 7534 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:57:22.556586 7534 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:57:22.563866 7534 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 22:57:22.564019 7534 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:57:22.564124 7534 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:57:22.564237 7534 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 22:57:22.564327 7534 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 22:57:22.574373 7534 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:38859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.94.62:44503
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.62.keytab
--principal=kudu/127.25.94.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.62:44503
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.94.62
--webserver_port=36365
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 22:57:22.575775 7534 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:57:22.577122 7534 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:57:22.589468 7539 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:57:22.589977 7542 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:57:22.590001 7540 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:57:22.590592 7534 server_base.cc:1034] running on GCE node
I20260807 22:57:22.591423 7534 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:57:22.593048 7534 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:57:22.594277 7534 hybrid_clock.cc:648] HybridClock initialized: now 1786143442594231 us; error 60 us; skew 500 ppm
I20260807 22:57:22.594674 7534 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 22:57:22 dist-test-slave-z8x9 krb5kdc[6653](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143442, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:57:22.598016 7534 init.cc:377] Logged in from keytab as kudu/127.25.94.62@KRBTEST.COM (short username kudu)
I20260807 22:57:22.599539 7534 webserver.cc:460] Webserver started at http://127.25.94.62:36365/ using document root <none> and password file <none>
I20260807 22:57:22.600256 7534 fs_manager.cc:359] Metadata directory not provided
I20260807 22:57:22.600425 7534 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:57:22.606434 7534 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260807 22:57:22.609503 7555 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:57:22.611163 7534 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260807 22:57:22.611370 7534 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/minicluster-data/master-0/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/minicluster-data/master-0/wal
uuid: "27f5d6942f0544838999b7466df2b0aa"
format_stamp: "Formatted at 2026-08-07 22:57:19 on dist-test-slave-z8x9"
server_key: "8002477171e63b29a3d12aea1b195d6e"
server_key_iv: "2c4d0bd8b888ce9ce2f10a61e6d4a9b1"
server_key_version: "encryptionkey@0"
I20260807 22:57:22.612432 7534 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:57:22.658761 7534 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:57:22.665452 7534 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:57:22.665863 7534 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:57:22.686856 7534 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.62:44503
I20260807 22:57:22.686884 7639 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.62:44503 every 8 connection(s)
I20260807 22:57:22.688553 7534 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
I20260807 22:57:22.694442 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 7534
I20260807 22:57:22.695286 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.1:39569
--local_ip_for_outbound_sockets=127.25.94.1
--tserver_master_addrs=127.25.94.62:44503
--webserver_port=46779
--webserver_interface=127.25.94.1
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.1.keytab
--principal=kudu/127.25.94.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.20:38859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.25.94.1
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/krb5kdc/oryx_127.25.94.1.keytab with env {KRB5CCNAME=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 22:57:22.696162 7642 sys_catalog.cc:263] Verifying existing consensus state
I20260807 22:57:22.699512 7642 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27f5d6942f0544838999b7466df2b0aa: Bootstrap starting.
I20260807 22:57:22.724979 7642 log.cc:824] T 00000000000000000000000000000000 P 27f5d6942f0544838999b7466df2b0aa: Log is configured to *not* fsync() on all Append() calls
I20260807 22:57:22.736598 7642 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27f5d6942f0544838999b7466df2b0aa: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 22:57:22.737354 7642 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27f5d6942f0544838999b7466df2b0aa: Bootstrap complete.
I20260807 22:57:22.750936 7642 raft_consensus.cc:348] T 00000000000000000000000000000000 P 27f5d6942f0544838999b7466df2b0aa [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27f5d6942f0544838999b7466df2b0aa" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 44503 } }
I20260807 22:57:22.752128 7642 raft_consensus.cc:729] T 00000000000000000000000000000000 P 27f5d6942f0544838999b7466df2b0aa [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27f5d6942f0544838999b7466df2b0aa, State: Initialized, Role: FOLLOWER
I20260807 22:57:22.752851 7642 consensus_queue.cc:260] T 00000000000000000000000000000000 P 27f5d6942f0544838999b7466df2b0aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27f5d6942f0544838999b7466df2b0aa" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 44503 } }
I20260807 22:57:22.753149 7642 raft_consensus.cc:388] T 00000000000000000000000000000000 P 27f5d6942f0544838999b7466df2b0aa [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 22:57:22.753309 7642 raft_consensus.cc:482] T 00000000000000000000000000000000 P 27f5d6942f0544838999b7466df2b0aa [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 22:57:22.753485 7642 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 27f5d6942f0544838999b7466df2b0aa [term 1 FOLLOWER]: Advancing to term 2
I20260807 22:57:22.759462 7642 raft_consensus.cc:504] T 00000000000000000000000000000000 P 27f5d6942f0544838999b7466df2b0aa [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27f5d6942f0544838999b7466df2b0aa" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 44503 } }
I20260807 22:57:22.760058 7642 leader_election.cc:302] T 00000000000000000000000000000000 P 27f5d6942f0544838999b7466df2b0aa [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 27f5d6942f0544838999b7466df2b0aa; no voters:
I20260807 22:57:22.760798 7642 leader_election.cc:288] T 00000000000000000000000000000000 P 27f5d6942f0544838999b7466df2b0aa [CANDIDATE]: Term 2 election: Requested vote from peers
I20260807 22:57:22.760943 7647 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 27f5d6942f0544838999b7466df2b0aa [term 2 FOLLOWER]: Leader election won for term 2
I20260807 22:57:22.762264 7647 raft_consensus.cc:686] T 00000000000000000000000000000000 P 27f5d6942f0544838999b7466df2b0aa [term 2 LEADER]: Becoming Leader. State: Replica: 27f5d6942f0544838999b7466df2b0aa, State: Running, Role: LEADER
I20260807 22:57:22.762885 7647 consensus_queue.cc:237] T 00000000000000000000000000000000 P 27f5d6942f0544838999b7466df2b0aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27f5d6942f0544838999b7466df2b0aa" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 44503 } }
I20260807 22:57:22.763108 7642 sys_catalog.cc:564] T 00000000000000000000000000000000 P 27f5d6942f0544838999b7466df2b0aa [sys.catalog]: configured and running, proceeding with master startup.
I20260807 22:57:22.765354 7649 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27f5d6942f0544838999b7466df2b0aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 27f5d6942f0544838999b7466df2b0aa. Latest consensus state: current_term: 2 leader_uuid: "27f5d6942f0544838999b7466df2b0aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27f5d6942f0544838999b7466df2b0aa" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 44503 } } }
I20260807 22:57:22.765751 7649 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27f5d6942f0544838999b7466df2b0aa [sys.catalog]: This master's current role is: LEADER
I20260807 22:57:22.766244 7648 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27f5d6942f0544838999b7466df2b0aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "27f5d6942f0544838999b7466df2b0aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27f5d6942f0544838999b7466df2b0aa" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 44503 } } }
I20260807 22:57:22.766577 7648 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27f5d6942f0544838999b7466df2b0aa [sys.catalog]: This master's current role is: LEADER
I20260807 22:57:22.776926 7659 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 22:57:22.783929 7659 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 22:57:22.785559 7659 catalog_manager.cc:1259] Loaded cluster ID: 8028894b86714d738551c09cae84a81d
I20260807 22:57:22.785744 7659 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 22:57:22.789976 7659 catalog_manager.cc:1504] Loading token signing keys...
I20260807 22:57:22.791585 7659 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 27f5d6942f0544838999b7466df2b0aa: Loaded TSK: 0
I20260807 22:57:22.793292 7659 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260807 22:57:23.074559 7645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:57:23.074842 7645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:57:23.074993 7645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:57:23.082780 7645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:57:23.082908 7645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:57:23.083093 7645 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.1
I20260807 22:57:23.093458 7645 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:38859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/krb5kdc/oryx_127.25.94.1.keytab
--principal=oryx/127.25.94.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.1:39569
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.94.1
--webserver_port=46779
--tserver_master_addrs=127.25.94.62:44503
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.1
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 22:57:23.094993 7645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:57:23.096349 7645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:57:23.115140 7691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:57:23.115384 7686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:57:23.115237 7685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:57:23.116534 7645 server_base.cc:1034] running on GCE node
I20260807 22:57:23.117429 7645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:57:23.118584 7645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:57:23.119839 7645 hybrid_clock.cc:648] HybridClock initialized: now 1786143443119798 us; error 45 us; skew 500 ppm
I20260807 22:57:23.120198 7645 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 22:57:23 dist-test-slave-z8x9 krb5kdc[6653](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143443, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.94.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:57:23.124035 7645 init.cc:377] Logged in from keytab as oryx/127.25.94.1@KRBTEST.COM (short username oryx)
I20260807 22:57:23.125939 7645 webserver.cc:460] Webserver started at http://127.25.94.1:46779/ using document root <none> and password file <none>
I20260807 22:57:23.126717 7645 fs_manager.cc:359] Metadata directory not provided
I20260807 22:57:23.126875 7645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:57:23.133353 7645 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.007s sys 0.000s
I20260807 22:57:23.136545 7701 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:57:23.138310 7645 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260807 22:57:23.138533 7645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/minicluster-data/ts-0/wal
uuid: "060a4bf42b3d4eb3a8855c021faee130"
format_stamp: "Formatted at 2026-08-07 22:57:20 on dist-test-slave-z8x9"
server_key: "cafe2402d1022765e3694f1c8c7b8c09"
server_key_iv: "5b6873e15de3f1641ddbc8cf801a90e6"
server_key_version: "encryptionkey@0"
I20260807 22:57:23.139602 7645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:57:23.156638 7645 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:57:23.162168 7645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:57:23.162500 7645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:57:23.163666 7645 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:57:23.165529 7645 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:57:23.165681 7645 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:57:23.165807 7645 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:57:23.165900 7645 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:57:23.203249 7645 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.1:39569
I20260807 22:57:23.203312 7882 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.1:39569 every 8 connection(s)
I20260807 22:57:23.205874 7645 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/minicluster-data/ts-0/data/info.pb
I20260807 22:57:23.207962 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 7645
I20260807 22:57:23.208652 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.2:39931
--local_ip_for_outbound_sockets=127.25.94.2
--tserver_master_addrs=127.25.94.62:44503
--webserver_port=37755
--webserver_interface=127.25.94.2
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.2.keytab
--principal=kudu/127.25.94.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.20:38859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.25.94.2
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/krb5kdc/oryx_127.25.94.2.keytab with env {KRB5CCNAME=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 07 22:57:23 dist-test-slave-z8x9 krb5kdc[6653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.1@KRBTEST.COM for oryx/127.25.94.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 22:57:23 dist-test-slave-z8x9 krb5kdc[6653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.1@KRBTEST.COM for oryx/127.25.94.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 22:57:23.229115 7887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:23.210899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:44503 (local address 127.25.94.1:49943)
0807 22:57:23.211892 (+ 993us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:23.211937 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0807 22:57:23.213826 (+ 1889us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:57:23.215341 (+ 1515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:57:23.215354 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:57:23.216095 (+ 741us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:57:23.216793 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:23.216813 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:23.219636 (+ 2823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:23.219640 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:23.220931 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:23.220946 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:23.222155 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:23.222163 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:23.222442 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:23.223286 (+ 844us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:57:23.223318 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:57:23.227701 (+ 4383us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.94.62:44503: Server oryx/127.25.94.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":728,"thread_start_us":499,"threads_started":1}
W20260807 22:57:23.229142 7886 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:57:23.210720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.1:49943 (local address 127.25.94.62:44503)
0807 22:57:23.211542 (+ 822us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:23.211561 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:23.212100 (+ 539us) server_negotiation.cc:407] Connection header received
0807 22:57:23.214428 (+ 2328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:23.214441 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:23.214798 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:23.215186 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:23.217056 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:23.219424 (+ 2368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:23.221102 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:23.221883 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:23.222113 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:23.228107 (+ 5994us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.94.1:49943: BlockingRecv error: recv got EOF from 127.25.94.1:49943 (error 108)
Metrics: {"server-negotiator.queue_time_us":532,"thread_start_us":319,"threads_started":1}
W20260807 22:57:23.230103 7883 heartbeater.cc:644] Failed to heartbeat to 127.25.94.62:44503 (0 consecutive failures): Not authorized: Failed to ping master at 127.25.94.62:44503: Client connection negotiation failed: client connection to 127.25.94.62:44503: Server oryx/127.25.94.62@KRBTEST.COM not found in Kerberos database
Aug 07 22:57:23 dist-test-slave-z8x9 krb5kdc[6653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.1@KRBTEST.COM for oryx/127.25.94.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 22:57:23 dist-test-slave-z8x9 krb5kdc[6653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.1@KRBTEST.COM for oryx/127.25.94.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 22:57:23.241065 7887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:23.230841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:44503 (local address 127.25.94.1:54521)
0807 22:57:23.231030 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:23.231048 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 22:57:23.231183 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:57:23.231493 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:57:23.231497 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:57:23.231858 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:57:23.232003 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:23.232011 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:23.233060 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:23.233065 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:23.234047 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:23.234059 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:23.234774 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:23.234779 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:23.235108 (+ 329us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:23.236010 (+ 902us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:57:23.236041 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:57:23.240758 (+ 4717us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.94.62:44503: Server oryx/127.25.94.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":58}
W20260807 22:57:23.241544 7886 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:57:23.231005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.1:54521 (local address 127.25.94.62:44503)
0807 22:57:23.231191 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:23.231197 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:23.231219 (+ 22us) server_negotiation.cc:407] Connection header received
0807 22:57:23.231286 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:23.231291 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:23.231364 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:23.231467 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:23.232168 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:23.232915 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:23.234197 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:23.234612 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:23.234746 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:23.241023 (+ 6277us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.94.1:54521: BlockingRecv error: recv got EOF from 127.25.94.1:54521 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Aug 07 22:57:23 dist-test-slave-z8x9 krb5kdc[6653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.1@KRBTEST.COM for oryx/127.25.94.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 22:57:23 dist-test-slave-z8x9 krb5kdc[6653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.1@KRBTEST.COM for oryx/127.25.94.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 22:57:23.251489 7887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:23.241925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:44503 (local address 127.25.94.1:60931)
0807 22:57:23.242104 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:23.242124 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 22:57:23.242261 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:57:23.242630 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:57:23.242635 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:57:23.243048 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:57:23.243268 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:23.243279 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:23.244407 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:23.244413 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:23.246037 (+ 1624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:23.246061 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:23.246890 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:23.246897 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:23.247119 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:23.247835 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:57:23.247860 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:57:23.251254 (+ 3394us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.94.62:44503: Server oryx/127.25.94.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":44}
W20260807 22:57:23.251875 7883 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260807 22:57:23.252007 7886 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:57:23.242019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.1:60931 (local address 127.25.94.62:44503)
0807 22:57:23.242178 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:23.242184 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:23.242203 (+ 19us) server_negotiation.cc:407] Connection header received
0807 22:57:23.242400 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:23.242406 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:23.242494 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:23.242601 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:23.243422 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:23.244205 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:23.246225 (+ 2020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:23.246685 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:23.246814 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:23.251564 (+ 4750us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.94.1:60931: BlockingRecv error: recv got EOF from 127.25.94.1:60931 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260807 22:57:23.559623 7888 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:57:23.559931 7888 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:57:23.560029 7888 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:57:23.567628 7888 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:57:23.567765 7888 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:57:23.567883 7888 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.2
I20260807 22:57:23.577755 7888 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:38859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/krb5kdc/oryx_127.25.94.2.keytab
--principal=oryx/127.25.94.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.2:39931
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.94.2
--webserver_port=37755
--tserver_master_addrs=127.25.94.62:44503
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.2
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 22:57:23.579092 7888 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:57:23.580372 7888 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:57:23.594252 7895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:57:23.594290 7897 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:57:23.594846 7901 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:57:23.595189 7888 server_base.cc:1034] running on GCE node
I20260807 22:57:23.595813 7888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:57:23.596827 7888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:57:23.598105 7888 hybrid_clock.cc:648] HybridClock initialized: now 1786143443597988 us; error 85 us; skew 500 ppm
I20260807 22:57:23.598479 7888 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 22:57:23 dist-test-slave-z8x9 krb5kdc[6653](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143443, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.94.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:57:23.601809 7888 init.cc:377] Logged in from keytab as oryx/127.25.94.2@KRBTEST.COM (short username oryx)
I20260807 22:57:23.603229 7888 webserver.cc:460] Webserver started at http://127.25.94.2:37755/ using document root <none> and password file <none>
I20260807 22:57:23.603981 7888 fs_manager.cc:359] Metadata directory not provided
I20260807 22:57:23.604135 7888 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:57:23.609637 7888 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260807 22:57:23.612272 7921 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:57:23.613672 7888 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 22:57:23.613854 7888 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/minicluster-data/ts-1/wal
uuid: "6c98b9c6759544f2bb23b4534d83c837"
format_stamp: "Formatted at 2026-08-07 22:57:20 on dist-test-slave-z8x9"
server_key: "15fa569ffd76bd432f418e9376591901"
server_key_iv: "b81ac75a99d87569d8193f7fe5d5efbc"
server_key_version: "encryptionkey@0"
I20260807 22:57:23.614861 7888 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:57:23.636739 7888 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:57:23.641475 7888 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:57:23.641763 7888 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:57:23.642800 7888 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:57:23.644335 7888 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:57:23.644461 7888 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:57:23.644586 7888 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:57:23.644680 7888 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:57:23.681613 7888 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.2:39931
I20260807 22:57:23.681703 8124 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.2:39931 every 8 connection(s)
I20260807 22:57:23.683127 7888 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/minicluster-data/ts-1/data/info.pb
I20260807 22:57:23.688220 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 7888
I20260807 22:57:23.688923 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.3:44957
--local_ip_for_outbound_sockets=127.25.94.3
--tserver_master_addrs=127.25.94.62:44503
--webserver_port=35477
--webserver_interface=127.25.94.3
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.3.keytab
--principal=kudu/127.25.94.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.20:38859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.25.94.3
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/krb5kdc/oryx_127.25.94.3.keytab with env {KRB5CCNAME=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 07 22:57:23 dist-test-slave-z8x9 krb5kdc[6653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.2@KRBTEST.COM for oryx/127.25.94.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 22:57:23 dist-test-slave-z8x9 krb5kdc[6653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.2@KRBTEST.COM for oryx/127.25.94.62@KRBTEST.COM, Server not found in Kerberos database
W20260807 22:57:23.700489 7886 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:57:23.685986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.2:52823 (local address 127.25.94.62:44503)
0807 22:57:23.686217 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:23.686224 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:23.687553 (+ 1329us) server_negotiation.cc:407] Connection header received
0807 22:57:23.689699 (+ 2146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:23.689705 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:23.689807 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:23.689926 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:23.691460 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:23.692184 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:23.693851 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:23.694317 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:23.694458 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:23.700215 (+ 5757us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.94.2:52823: BlockingRecv error: recv got EOF from 127.25.94.2:52823 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
I20260807 22:57:23.701488 8129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:23.686524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:44503 (local address 127.25.94.2:52823)
0807 22:57:23.687379 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:23.687424 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0807 22:57:23.689407 (+ 1983us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:57:23.690174 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:57:23.690182 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:57:23.690710 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:57:23.691267 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:23.691284 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:23.692374 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:23.692378 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:23.693696 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:23.693709 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:23.694566 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:23.694573 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:23.694807 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:23.695544 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:57:23.695573 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:57:23.699967 (+ 4394us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.94.62:44503: Server oryx/127.25.94.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":376,"threads_started":1}
W20260807 22:57:23.702245 8127 heartbeater.cc:644] Failed to heartbeat to 127.25.94.62:44503 (0 consecutive failures): Not authorized: Failed to ping master at 127.25.94.62:44503: Client connection negotiation failed: client connection to 127.25.94.62:44503: Server oryx/127.25.94.62@KRBTEST.COM not found in Kerberos database
Aug 07 22:57:23 dist-test-slave-z8x9 krb5kdc[6653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.2@KRBTEST.COM for oryx/127.25.94.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 22:57:23 dist-test-slave-z8x9 krb5kdc[6653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.2@KRBTEST.COM for oryx/127.25.94.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 22:57:23.712570 8129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:23.702926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:44503 (local address 127.25.94.2:50011)
0807 22:57:23.703201 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:23.703223 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 22:57:23.703343 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:57:23.703673 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:57:23.703680 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:57:23.704156 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:57:23.704393 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:23.704409 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:23.705475 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:23.705484 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:23.707152 (+ 1668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:23.707170 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:23.707879 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:23.707887 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:23.708107 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:23.708777 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:57:23.708809 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:57:23.712265 (+ 3456us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.94.62:44503: Server oryx/127.25.94.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":140}
W20260807 22:57:23.712728 7886 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:57:23.703064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.2:50011 (local address 127.25.94.62:44503)
0807 22:57:23.703226 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:23.703230 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:23.703276 (+ 46us) server_negotiation.cc:407] Connection header received
0807 22:57:23.703459 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:23.703463 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:23.703522 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:23.703606 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:23.704538 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:23.705329 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:23.707312 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:23.707712 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:23.707816 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:23.712487 (+ 4671us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.94.2:50011: BlockingRecv error: recv got EOF from 127.25.94.2:50011 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Aug 07 22:57:23 dist-test-slave-z8x9 krb5kdc[6653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.2@KRBTEST.COM for oryx/127.25.94.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 22:57:23 dist-test-slave-z8x9 krb5kdc[6653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.2@KRBTEST.COM for oryx/127.25.94.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 22:57:23.721501 8129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:23.713521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:44503 (local address 127.25.94.2:49049)
0807 22:57:23.713694 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:23.713713 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 22:57:23.713812 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:57:23.714114 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:57:23.714119 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:57:23.714434 (+ 315us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:57:23.714572 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:23.714580 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:23.715589 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:23.715594 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:23.716579 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:23.716590 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:23.717457 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:23.717463 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:23.717643 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:23.718234 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:57:23.718257 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:57:23.721154 (+ 2897us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.94.62:44503: Server oryx/127.25.94.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":59}
W20260807 22:57:23.721738 7886 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:57:23.713644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.2:49049 (local address 127.25.94.62:44503)
0807 22:57:23.713834 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:23.713842 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:23.713864 (+ 22us) server_negotiation.cc:407] Connection header received
0807 22:57:23.713916 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:23.713921 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:23.713988 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:23.714091 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:23.714710 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:23.715451 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:23.716717 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:23.717301 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:23.717434 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:23.721396 (+ 3962us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.94.2:49049: BlockingRecv error: recv got EOF from 127.25.94.2:49049 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260807 22:57:23.721891 8127 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260807 22:57:24.034713 8130 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:57:24.035032 8130 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:57:24.035130 8130 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:57:24.042933 8130 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:57:24.043062 8130 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:57:24.043179 8130 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.3
I20260807 22:57:24.053316 8130 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:38859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/krb5kdc/oryx_127.25.94.3.keytab
--principal=oryx/127.25.94.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.3:44957
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.94.3
--webserver_port=35477
--tserver_master_addrs=127.25.94.62:44503
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.3
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 22:57:24.054703 8130 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:57:24.055948 8130 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:57:24.069645 8136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:57:24.069718 8140 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:57:24.069617 8137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:57:24.070411 8130 server_base.cc:1034] running on GCE node
I20260807 22:57:24.071027 8130 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:57:24.072016 8130 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:57:24.073319 8130 hybrid_clock.cc:648] HybridClock initialized: now 1786143444073221 us; error 81 us; skew 500 ppm
I20260807 22:57:24.073905 8130 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 22:57:24 dist-test-slave-z8x9 krb5kdc[6653](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143444, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.94.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:57:24.077648 8130 init.cc:377] Logged in from keytab as oryx/127.25.94.3@KRBTEST.COM (short username oryx)
I20260807 22:57:24.079298 8130 webserver.cc:460] Webserver started at http://127.25.94.3:35477/ using document root <none> and password file <none>
I20260807 22:57:24.080031 8130 fs_manager.cc:359] Metadata directory not provided
I20260807 22:57:24.080210 8130 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:57:24.086120 8130 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260807 22:57:24.089020 8158 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:57:24.090569 8130 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260807 22:57:24.090763 8130 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/minicluster-data/ts-2/wal
uuid: "613b6c8ff6d54551931c538ce7d9d9b1"
format_stamp: "Formatted at 2026-08-07 22:57:21 on dist-test-slave-z8x9"
server_key: "47e27998c80c1bcb028f34bdf792e764"
server_key_iv: "25483f3b839cd93e42f5a84cb5d5b3f7"
server_key_version: "encryptionkey@0"
I20260807 22:57:24.091790 8130 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:57:24.113240 8130 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:57:24.118458 8130 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:57:24.118780 8130 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:57:24.120021 8130 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:57:24.121639 8130 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:57:24.121764 8130 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:57:24.121920 8130 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:57:24.122020 8130 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:57:24.159574 8130 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.3:44957
I20260807 22:57:24.159639 8353 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.3:44957 every 8 connection(s)
I20260807 22:57:24.161110 8130 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786143361779964-25976-0/minicluster-data/ts-2/data/info.pb
I20260807 22:57:24.170488 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 8130
Aug 07 22:57:24 dist-test-slave-z8x9 krb5kdc[6653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.3@KRBTEST.COM for oryx/127.25.94.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 22:57:24 dist-test-slave-z8x9 krb5kdc[6653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.3@KRBTEST.COM for oryx/127.25.94.62@KRBTEST.COM, Server not found in Kerberos database
W20260807 22:57:24.181054 7886 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:57:24.164026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.3:57717 (local address 127.25.94.62:44503)
0807 22:57:24.164275 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:24.164282 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:24.165747 (+ 1465us) server_negotiation.cc:407] Connection header received
0807 22:57:24.167688 (+ 1941us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:24.167694 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:24.167768 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:24.167909 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:24.170060 (+ 2151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:24.171198 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:24.173724 (+ 2526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:24.174166 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:24.174265 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:24.180763 (+ 6498us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.94.3:57717: BlockingRecv error: recv got EOF from 127.25.94.3:57717 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
I20260807 22:57:24.181928 8359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:24.164658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:44503 (local address 127.25.94.3:57717)
0807 22:57:24.165569 (+ 911us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:24.165608 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0807 22:57:24.167362 (+ 1754us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:57:24.168219 (+ 857us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:57:24.168234 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:57:24.169106 (+ 872us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:57:24.169787 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:24.169806 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:24.171465 (+ 1659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:24.171470 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:24.173553 (+ 2083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:24.173569 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:24.174447 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:24.174456 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:24.174765 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:24.175544 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:57:24.175573 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:57:24.180554 (+ 4981us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.94.62:44503: Server oryx/127.25.94.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":649,"thread_start_us":306,"threads_started":1}
W20260807 22:57:24.182629 8356 heartbeater.cc:644] Failed to heartbeat to 127.25.94.62:44503 (0 consecutive failures): Not authorized: Failed to ping master at 127.25.94.62:44503: Client connection negotiation failed: client connection to 127.25.94.62:44503: Server oryx/127.25.94.62@KRBTEST.COM not found in Kerberos database
I20260807 22:57:24.189419 8361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:24.171631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56604 (local address 127.25.94.62:44503)
0807 22:57:24.172515 (+ 884us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:24.172519 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:24.172535 (+ 16us) server_negotiation.cc:407] Connection header received
0807 22:57:24.172575 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:24.172579 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:24.172632 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:24.172720 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:24.174715 (+ 1995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:24.175570 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:24.177516 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:24.178221 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:24.178374 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:24.181430 (+ 3056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:57:24.181443 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:57:24.181468 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:57:24.181509 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:57:24.185249 (+ 3740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:24.186344 (+ 1095us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:24.186353 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:24.186361 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:24.186630 (+ 269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:24.187078 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:24.187085 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:24.187088 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:24.187682 (+ 594us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:57:24.188028 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0807 22:57:24.188547 (+ 519us) server_negotiation.cc:299] Negotiation successful
0807 22:57:24.189017 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":798,"thread_start_us":645,"threads_started":1}
Aug 07 22:57:24 dist-test-slave-z8x9 krb5kdc[6653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.3@KRBTEST.COM for oryx/127.25.94.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 22:57:24 dist-test-slave-z8x9 krb5kdc[6653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.3@KRBTEST.COM for oryx/127.25.94.62@KRBTEST.COM, Server not found in Kerberos database
W20260807 22:57:24.192795 7886 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:57:24.183408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.3:55805 (local address 127.25.94.62:44503)
0807 22:57:24.183644 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:24.183653 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:24.183676 (+ 23us) server_negotiation.cc:407] Connection header received
0807 22:57:24.183897 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:24.183901 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:24.183960 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:24.184094 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:24.184796 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:24.185628 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:24.187053 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:24.187552 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:24.187718 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:24.192598 (+ 4880us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.94.3:55805: BlockingRecv error: recv got EOF from 127.25.94.3:55805 (error 108)
Metrics: {"server-negotiator.queue_time_us":116}
I20260807 22:57:24.193125 8359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:24.183363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:44503 (local address 127.25.94.3:55805)
0807 22:57:24.183597 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:24.183617 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 22:57:24.183737 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:57:24.184115 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:57:24.184119 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:57:24.184497 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:57:24.184645 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:24.184654 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:24.185774 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:24.185780 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:24.186909 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:24.186923 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:24.187751 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:24.187757 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:24.187933 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:24.188555 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:57:24.188574 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:57:24.192364 (+ 3790us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.94.62:44503: Server oryx/127.25.94.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":74}
Aug 07 22:57:24 dist-test-slave-z8x9 krb5kdc[6653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.3@KRBTEST.COM for oryx/127.25.94.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 22:57:24 dist-test-slave-z8x9 krb5kdc[6653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.3@KRBTEST.COM for oryx/127.25.94.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 22:57:24.202633 8359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:24.194221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:44503 (local address 127.25.94.3:45687)
0807 22:57:24.194395 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:24.194413 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 22:57:24.194506 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:57:24.194856 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:57:24.194860 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:57:24.195174 (+ 314us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:57:24.195339 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:24.195350 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:24.196485 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:24.196489 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:24.197556 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:24.197569 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:24.198259 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:24.198265 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:24.198430 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:24.199308 (+ 878us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:57:24.199327 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:57:24.202322 (+ 2995us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.94.62:44503: Server oryx/127.25.94.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":58}
W20260807 22:57:24.203032 8356 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260807 22:57:24.203423 7886 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:57:24.194327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.3:45687 (local address 127.25.94.62:44503)
0807 22:57:24.194539 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:24.194544 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:24.194561 (+ 17us) server_negotiation.cc:407] Connection header received
0807 22:57:24.194606 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:24.194610 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:24.194668 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:24.194748 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:24.195523 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:24.196323 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:24.197702 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:24.198097 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:24.198198 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:24.202676 (+ 4478us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.94.3:45687: BlockingRecv error: recv got EOF from 127.25.94.3:45687 (error 108)
Metrics: {"server-negotiator.queue_time_us":112}
Aug 07 22:57:24 dist-test-slave-z8x9 krb5kdc[6653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.1@KRBTEST.COM for oryx/127.25.94.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 22:57:24 dist-test-slave-z8x9 krb5kdc[6653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.1@KRBTEST.COM for oryx/127.25.94.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 22:57:24.261644 8363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:24.252628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:44503 (local address 127.25.94.1:48233)
0807 22:57:24.253248 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:24.253270 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 22:57:24.253392 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:57:24.253712 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:57:24.253715 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:57:24.254247 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:57:24.254523 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:24.254541 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:24.255827 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:24.255833 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:24.257035 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:24.257049 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:24.257811 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:24.257816 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:24.257938 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:24.258536 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:57:24.258554 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:57:24.261413 (+ 2859us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.94.62:44503: Server oryx/127.25.94.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":318,"threads_started":1}
W20260807 22:57:24.261858 7886 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:57:24.252722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.1:48233 (local address 127.25.94.62:44503)
0807 22:57:24.252923 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:24.252927 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:24.253350 (+ 423us) server_negotiation.cc:407] Connection header received
0807 22:57:24.253507 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:24.253512 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:24.253577 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:24.253669 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:24.254705 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:24.255662 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:24.257204 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:24.257635 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:24.257734 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:24.261643 (+ 3909us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.94.1:48233: BlockingRecv error: recv got EOF from 127.25.94.1:48233 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Aug 07 22:57:24 dist-test-slave-z8x9 krb5kdc[6653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.2@KRBTEST.COM for oryx/127.25.94.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 22:57:24 dist-test-slave-z8x9 krb5kdc[6653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.2@KRBTEST.COM for oryx/127.25.94.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 22:57:24.731417 8364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:24.722601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:44503 (local address 127.25.94.2:59337)
0807 22:57:24.723154 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:24.723171 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 22:57:24.723288 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:57:24.723711 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:57:24.723716 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:57:24.724465 (+ 749us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:57:24.724696 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:24.724706 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:24.725796 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:24.725800 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:24.726797 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:24.726810 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:24.727493 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:24.727498 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:24.727614 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:24.728290 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:57:24.728308 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:57:24.731179 (+ 2871us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.94.62:44503: Server oryx/127.25.94.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":290,"threads_started":1}
W20260807 22:57:24.731779 7886 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:57:24.722658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.2:59337 (local address 127.25.94.62:44503)
0807 22:57:24.722891 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:24.722897 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:24.723293 (+ 396us) server_negotiation.cc:407] Connection header received
0807 22:57:24.723445 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:24.723451 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:24.723539 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:24.723634 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:24.724825 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:24.725625 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:24.726934 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:24.727345 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:24.727436 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:24.731418 (+ 3982us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.94.2:59337: BlockingRecv error: recv got EOF from 127.25.94.2:59337 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Aug 07 22:57:25 dist-test-slave-z8x9 krb5kdc[6653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.3@KRBTEST.COM for oryx/127.25.94.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 22:57:25 dist-test-slave-z8x9 krb5kdc[6653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.3@KRBTEST.COM for oryx/127.25.94.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 22:57:25.213321 8365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:25.203728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:44503 (local address 127.25.94.3:41193)
0807 22:57:25.204266 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:25.204287 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 22:57:25.204508 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:57:25.204872 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:57:25.204876 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:57:25.205481 (+ 605us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:57:25.205679 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:25.205688 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:25.206783 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:25.206789 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:25.208042 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:25.208057 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:25.208858 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:25.208864 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:25.209049 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:25.209785 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:57:25.209807 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:57:25.212936 (+ 3129us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.94.62:44503: Server oryx/127.25.94.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":271,"threads_started":1}
W20260807 22:57:25.213496 7886 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:57:25.203807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.3:41193 (local address 127.25.94.62:44503)
0807 22:57:25.204022 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:25.204032 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:25.204373 (+ 341us) server_negotiation.cc:407] Connection header received
0807 22:57:25.204654 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:25.204659 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:25.204734 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:25.204842 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:25.205839 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:25.206613 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:25.208217 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:25.208637 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:25.208809 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:25.213242 (+ 4433us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.94.3:41193: BlockingRecv error: recv got EOF from 127.25.94.3:41193 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Aug 07 22:57:25 dist-test-slave-z8x9 krb5kdc[6653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.1@KRBTEST.COM for oryx/127.25.94.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 22:57:25 dist-test-slave-z8x9 krb5kdc[6653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.1@KRBTEST.COM for oryx/127.25.94.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 22:57:25.271915 8368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:25.262826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:44503 (local address 127.25.94.1:46721)
0807 22:57:25.263447 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:25.263473 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0807 22:57:25.263608 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:57:25.264018 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:57:25.264021 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:57:25.264451 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:57:25.264597 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:25.264605 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:25.265889 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:25.265894 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:25.266874 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:25.266886 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:25.267870 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:25.267882 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:25.268001 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:25.268683 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:57:25.268700 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:57:25.271654 (+ 2954us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.94.62:44503: Server oryx/127.25.94.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":340,"threads_started":1}
W20260807 22:57:25.272241 7886 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:57:25.262946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.1:46721 (local address 127.25.94.62:44503)
0807 22:57:25.263119 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:25.263123 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:25.263577 (+ 454us) server_negotiation.cc:407] Connection header received
0807 22:57:25.263768 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:25.263779 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:25.263864 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:25.263961 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:25.264757 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:25.265571 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:25.267034 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:25.267649 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:25.267797 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:25.271916 (+ 4119us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.94.1:46721: BlockingRecv error: recv got EOF from 127.25.94.1:46721 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Aug 07 22:57:25 dist-test-slave-z8x9 krb5kdc[6653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.2@KRBTEST.COM for oryx/127.25.94.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 22:57:25 dist-test-slave-z8x9 krb5kdc[6653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.2@KRBTEST.COM for oryx/127.25.94.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 22:57:25.741525 8369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:25.732420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:44503 (local address 127.25.94.2:53991)
0807 22:57:25.733053 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:25.733071 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 22:57:25.733316 (+ 245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:57:25.733698 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:57:25.733702 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:57:25.734189 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:57:25.734335 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:25.734344 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:25.735366 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:25.735379 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:25.736305 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:25.736317 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:25.737091 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:25.737098 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:25.737323 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:25.737960 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:57:25.737978 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:57:25.741128 (+ 3150us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.94.62:44503: Server oryx/127.25.94.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":336,"threads_started":1}
W20260807 22:57:25.741673 7886 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:57:25.732547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.2:53991 (local address 127.25.94.62:44503)
0807 22:57:25.732789 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:25.732797 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:25.733198 (+ 401us) server_negotiation.cc:407] Connection header received
0807 22:57:25.733455 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:25.733461 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:25.733555 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:25.733680 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:25.734480 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:25.735219 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:25.736449 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:25.736916 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:25.737054 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:25.741419 (+ 4365us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.94.2:53991: BlockingRecv error: recv got EOF from 127.25.94.2:53991 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Aug 07 22:57:26 dist-test-slave-z8x9 krb5kdc[6653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.3@KRBTEST.COM for oryx/127.25.94.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 22:57:26 dist-test-slave-z8x9 krb5kdc[6653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.3@KRBTEST.COM for oryx/127.25.94.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 22:57:26.223994 8371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:26.214714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:44503 (local address 127.25.94.3:57341)
0807 22:57:26.215339 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:26.215356 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 22:57:26.215487 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:57:26.215905 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:57:26.215908 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:57:26.216345 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:57:26.216496 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:26.216505 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:26.217995 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:26.218001 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:26.219121 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:26.219135 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:26.219921 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:26.219926 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:26.220043 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:26.220711 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:57:26.220731 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:57:26.223725 (+ 2994us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.94.62:44503: Server oryx/127.25.94.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":357,"threads_started":1}
W20260807 22:57:26.224257 7886 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:57:26.214860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.3:57341 (local address 127.25.94.62:44503)
0807 22:57:26.215049 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:26.215053 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:26.215481 (+ 428us) server_negotiation.cc:407] Connection header received
0807 22:57:26.215621 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:26.215628 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:26.215725 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:26.215836 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:26.216695 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:26.217754 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:26.219294 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:26.219754 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:26.219871 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:26.223979 (+ 4108us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.94.3:57341: BlockingRecv error: recv got EOF from 127.25.94.3:57341 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Aug 07 22:57:26 dist-test-slave-z8x9 krb5kdc[6653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.1@KRBTEST.COM for oryx/127.25.94.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 22:57:26 dist-test-slave-z8x9 krb5kdc[6653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.1@KRBTEST.COM for oryx/127.25.94.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 22:57:26.287004 8372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:26.272976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:44503 (local address 127.25.94.1:53191)
0807 22:57:26.273585 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:26.273607 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 22:57:26.273773 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:57:26.274131 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:57:26.274136 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:57:26.274579 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:57:26.274815 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:26.274827 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:26.276240 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:26.276247 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:26.277322 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:26.277335 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:26.278070 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:26.278076 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:26.278268 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:26.283480 (+ 5212us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:57:26.283503 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:57:26.286670 (+ 3167us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.94.62:44503: Server oryx/127.25.94.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":372,"threads_started":1}
W20260807 22:57:26.287386 7886 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:57:26.273123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.1:53191 (local address 127.25.94.62:44503)
0807 22:57:26.273429 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:26.273437 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:26.273694 (+ 257us) server_negotiation.cc:407] Connection header received
0807 22:57:26.273906 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:26.273911 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:26.273990 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:26.274098 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:26.274974 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:26.276084 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:26.277466 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:26.277916 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:26.278055 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:26.287010 (+ 8955us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.94.1:53191: BlockingRecv error: recv got EOF from 127.25.94.1:53191 (error 108)
Metrics: {"server-negotiator.queue_time_us":151}
Aug 07 22:57:26 dist-test-slave-z8x9 krb5kdc[6653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.2@KRBTEST.COM for oryx/127.25.94.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 22:57:26 dist-test-slave-z8x9 krb5kdc[6653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.2@KRBTEST.COM for oryx/127.25.94.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 22:57:26.751757 8373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:26.742781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:44503 (local address 127.25.94.2:38557)
0807 22:57:26.743359 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:26.743403 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0807 22:57:26.743625 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:57:26.744088 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:57:26.744091 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:57:26.744604 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:57:26.744803 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:26.744812 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:26.746017 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:26.746021 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:26.747028 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:26.747040 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:26.747858 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:26.747862 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:26.747994 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:26.748695 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:57:26.748714 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:57:26.751499 (+ 2785us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.94.62:44503: Server oryx/127.25.94.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":274,"threads_started":1}
W20260807 22:57:26.751969 7886 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:57:26.742878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.2:38557 (local address 127.25.94.62:44503)
0807 22:57:26.743092 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:26.743100 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:26.743536 (+ 436us) server_negotiation.cc:407] Connection header received
0807 22:57:26.743816 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:26.743822 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:26.743908 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:26.744037 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:26.744963 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:26.745871 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:26.747207 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:26.747691 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:26.747816 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:26.751734 (+ 3918us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.94.2:38557: BlockingRecv error: recv got EOF from 127.25.94.2:38557 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Aug 07 22:57:27 dist-test-slave-z8x9 krb5kdc[6653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.3@KRBTEST.COM for oryx/127.25.94.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 22:57:27 dist-test-slave-z8x9 krb5kdc[6653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.3@KRBTEST.COM for oryx/127.25.94.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 22:57:27.234064 8374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:27.225276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:44503 (local address 127.25.94.3:54297)
0807 22:57:27.225860 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:27.225878 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 22:57:27.225999 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:57:27.226369 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:57:27.226376 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:57:27.226752 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:57:27.226945 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:27.226954 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:27.228053 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:27.228057 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:27.229006 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:27.229021 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:27.229828 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:27.229833 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:27.229985 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:27.230682 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:57:27.230702 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:57:27.233791 (+ 3089us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.94.62:44503: Server oryx/127.25.94.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":330,"threads_started":1}
W20260807 22:57:27.234320 7886 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:57:27.225467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.3:54297 (local address 127.25.94.62:44503)
0807 22:57:27.225706 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:27.225715 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:27.225959 (+ 244us) server_negotiation.cc:407] Connection header received
0807 22:57:27.226099 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:27.226103 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:27.226175 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:27.226264 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:27.227106 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:27.227865 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:27.229166 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:27.229665 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:27.229785 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:27.234054 (+ 4269us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.94.3:54297: BlockingRecv error: recv got EOF from 127.25.94.3:54297 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Aug 07 22:57:27 dist-test-slave-z8x9 krb5kdc[6653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.1@KRBTEST.COM for oryx/127.25.94.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 22:57:27 dist-test-slave-z8x9 krb5kdc[6653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.1@KRBTEST.COM for oryx/127.25.94.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 22:57:27.296814 8375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:27.288093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:44503 (local address 127.25.94.1:58155)
0807 22:57:27.288586 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:27.288606 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 22:57:27.288719 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:57:27.289062 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:57:27.289066 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:57:27.289537 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:57:27.289691 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:27.289701 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:27.290739 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:27.290744 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:27.291773 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:27.291787 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:27.292615 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:27.292619 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:27.292768 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:27.293512 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:57:27.293530 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:57:27.296563 (+ 3033us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.94.62:44503: Server oryx/127.25.94.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":298,"threads_started":1}
W20260807 22:57:27.297331 7886 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:57:27.288141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.1:58155 (local address 127.25.94.62:44503)
0807 22:57:27.288439 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:27.288445 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:27.288689 (+ 244us) server_negotiation.cc:407] Connection header received
0807 22:57:27.288830 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:27.288834 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:27.288899 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:27.288982 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:27.289820 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:27.290595 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:27.291998 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:27.292450 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:27.292563 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:27.296938 (+ 4375us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.94.1:58155: BlockingRecv error: recv got EOF from 127.25.94.1:58155 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Aug 07 22:57:27 dist-test-slave-z8x9 krb5kdc[6653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.2@KRBTEST.COM for oryx/127.25.94.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 22:57:27 dist-test-slave-z8x9 krb5kdc[6653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.2@KRBTEST.COM for oryx/127.25.94.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 22:57:27.761989 8376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:27.752905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:44503 (local address 127.25.94.2:51469)
0807 22:57:27.753515 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:27.753538 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0807 22:57:27.753692 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:57:27.754116 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:57:27.754123 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:57:27.754680 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:57:27.754851 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:27.754860 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:27.755864 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:27.755868 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:27.756822 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:27.756834 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:27.757902 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:27.757906 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:27.758037 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:27.758737 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:57:27.758756 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:57:27.761737 (+ 2981us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.94.62:44503: Server oryx/127.25.94.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":324,"threads_started":1}
W20260807 22:57:27.762696 7886 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:57:27.753210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.2:51469 (local address 127.25.94.62:44503)
0807 22:57:27.753412 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:27.753419 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:27.753659 (+ 240us) server_negotiation.cc:407] Connection header received
0807 22:57:27.753844 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:27.753852 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:27.753943 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:27.754068 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:27.754985 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:27.755737 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:27.757007 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:27.757727 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:27.757835 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:27.762321 (+ 4486us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.94.2:51469: BlockingRecv error: recv got EOF from 127.25.94.2:51469 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Aug 07 22:57:28 dist-test-slave-z8x9 krb5kdc[6653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.3@KRBTEST.COM for oryx/127.25.94.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 22:57:28 dist-test-slave-z8x9 krb5kdc[6653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.3@KRBTEST.COM for oryx/127.25.94.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 22:57:28.245343 8379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:28.235371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:44503 (local address 127.25.94.3:46301)
0807 22:57:28.235977 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:28.235996 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 22:57:28.236122 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:57:28.236577 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:57:28.236581 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:57:28.237315 (+ 734us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:57:28.237492 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:28.237502 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:28.238823 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:28.238830 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:28.239994 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:28.240010 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:28.240782 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:28.240788 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:28.240986 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:28.241706 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:57:28.241726 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:57:28.245026 (+ 3300us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.94.62:44503: Server oryx/127.25.94.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":360,"threads_started":1}
W20260807 22:57:28.245882 7886 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:57:28.235607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.3:46301 (local address 127.25.94.62:44503)
0807 22:57:28.235896 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:28.235902 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:28.236080 (+ 178us) server_negotiation.cc:407] Connection header received
0807 22:57:28.236285 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:28.236290 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:28.236370 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:28.236476 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:28.237724 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:28.238672 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:28.240160 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:28.240609 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:28.240764 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:28.245607 (+ 4843us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.94.3:46301: BlockingRecv error: recv got EOF from 127.25.94.3:46301 (error 108)
Metrics: {"server-negotiator.queue_time_us":146}
Aug 07 22:57:28 dist-test-slave-z8x9 krb5kdc[6653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.1@KRBTEST.COM for oryx/127.25.94.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 22:57:28 dist-test-slave-z8x9 krb5kdc[6653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.1@KRBTEST.COM for oryx/127.25.94.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 22:57:28.306885 8380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:28.297970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:44503 (local address 127.25.94.1:51937)
0807 22:57:28.298543 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:28.298564 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 22:57:28.298735 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:57:28.299114 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:57:28.299119 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:57:28.299577 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:57:28.299736 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:28.299745 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:28.300815 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:28.300824 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:28.301827 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:28.301839 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:28.302682 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:28.302687 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:28.302862 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:28.303470 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:57:28.303490 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:57:28.306635 (+ 3145us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.94.62:44503: Server oryx/127.25.94.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":291,"threads_started":1}
W20260807 22:57:28.307103 7886 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:57:28.298066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.1:51937 (local address 127.25.94.62:44503)
0807 22:57:28.298350 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:28.298357 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:28.298665 (+ 308us) server_negotiation.cc:407] Connection header received
0807 22:57:28.298870 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:28.298878 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:28.298949 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:28.299068 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:28.299900 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:28.300668 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:28.301989 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:28.302518 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:28.302654 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:28.306894 (+ 4240us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.94.1:51937: BlockingRecv error: recv got EOF from 127.25.94.1:51937 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
Aug 07 22:57:28 dist-test-slave-z8x9 krb5kdc[6653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.2@KRBTEST.COM for oryx/127.25.94.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 22:57:28 dist-test-slave-z8x9 krb5kdc[6653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.2@KRBTEST.COM for oryx/127.25.94.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 22:57:28.773243 8383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:28.763126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:44503 (local address 127.25.94.2:42983)
0807 22:57:28.763673 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:28.763690 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 22:57:28.763839 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:57:28.764207 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:57:28.764211 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:57:28.764603 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:57:28.764770 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:28.764778 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:28.765905 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:28.765913 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:28.766860 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:28.766873 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:28.767696 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:28.767702 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:28.767833 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:28.768507 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:57:28.768539 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:57:28.772801 (+ 4262us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.94.62:44503: Server oryx/127.25.94.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":265,"threads_started":1}
W20260807 22:57:28.773376 7886 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:57:28.763352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.2:42983 (local address 127.25.94.62:44503)
0807 22:57:28.763601 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:28.763613 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:28.763802 (+ 189us) server_negotiation.cc:407] Connection header received
0807 22:57:28.764008 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:28.764014 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:28.764085 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:28.764166 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:28.764916 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:28.765725 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:28.767015 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:28.767526 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:28.767624 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:28.773061 (+ 5437us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.94.2:42983: BlockingRecv error: recv got EOF from 127.25.94.2:42983 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Aug 07 22:57:29 dist-test-slave-z8x9 krb5kdc[6653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.3@KRBTEST.COM for oryx/127.25.94.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 22:57:29 dist-test-slave-z8x9 krb5kdc[6653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.3@KRBTEST.COM for oryx/127.25.94.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 22:57:29.255801 8386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:29.246465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:44503 (local address 127.25.94.3:40983)
0807 22:57:29.246993 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:29.247011 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 22:57:29.247125 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:57:29.247456 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:57:29.247460 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:57:29.247884 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:57:29.248050 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:29.248058 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:29.249101 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:29.249105 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:29.250173 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:29.250186 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:29.250982 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:29.250987 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:29.251111 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:29.251905 (+ 794us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:57:29.251925 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:57:29.255398 (+ 3473us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.94.62:44503: Server oryx/127.25.94.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":274,"threads_started":1}
W20260807 22:57:29.255901 7886 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:57:29.246560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.3:40983 (local address 127.25.94.62:44503)
0807 22:57:29.246819 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:29.246826 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:29.247133 (+ 307us) server_negotiation.cc:407] Connection header received
0807 22:57:29.247219 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:29.247223 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:29.247287 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:29.247371 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:29.248207 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:29.248960 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:29.250336 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:29.250812 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:29.250930 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:29.255618 (+ 4688us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.94.3:40983: BlockingRecv error: recv got EOF from 127.25.94.3:40983 (error 108)
Metrics: {"server-negotiator.queue_time_us":105}
Aug 07 22:57:29 dist-test-slave-z8x9 krb5kdc[6653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.1@KRBTEST.COM for oryx/127.25.94.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 22:57:29 dist-test-slave-z8x9 krb5kdc[6653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.1@KRBTEST.COM for oryx/127.25.94.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 22:57:29.316457 8387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:29.307923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:44503 (local address 127.25.94.1:39521)
0807 22:57:29.308465 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:29.308486 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 22:57:29.308609 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:57:29.308915 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:57:29.308918 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:57:29.309420 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:57:29.309588 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:29.309598 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:29.310642 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:29.310647 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:29.311602 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:29.311614 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:29.312350 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:29.312355 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:29.312481 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:29.313112 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:57:29.313129 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:57:29.316083 (+ 2954us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.94.62:44503: Server oryx/127.25.94.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":352,"threads_started":1}
W20260807 22:57:29.316612 7886 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:57:29.308124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.1:39521 (local address 127.25.94.62:44503)
0807 22:57:29.308447 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:29.308456 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:29.308564 (+ 108us) server_negotiation.cc:407] Connection header received
0807 22:57:29.308712 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:29.308716 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:29.308783 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:29.308866 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:29.309729 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:29.310498 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:29.311732 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:29.312146 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:29.312240 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:29.316364 (+ 4124us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.94.1:39521: BlockingRecv error: recv got EOF from 127.25.94.1:39521 (error 108)
Metrics: {"server-negotiator.queue_time_us":186}
Aug 07 22:57:29 dist-test-slave-z8x9 krb5kdc[6653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.2@KRBTEST.COM for oryx/127.25.94.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 22:57:29 dist-test-slave-z8x9 krb5kdc[6653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.2@KRBTEST.COM for oryx/127.25.94.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 22:57:29.784386 8389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:29.774528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:44503 (local address 127.25.94.2:54307)
0807 22:57:29.775191 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:29.775213 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 22:57:29.775371 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:57:29.775819 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:57:29.775824 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:57:29.776342 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:57:29.776514 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:29.776523 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:29.777905 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:29.777909 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:29.779079 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:29.779093 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:29.780045 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:29.780055 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:29.780186 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:29.780998 (+ 812us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:57:29.781019 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:57:29.784147 (+ 3128us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.94.62:44503: Server oryx/127.25.94.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":390,"threads_started":1}
W20260807 22:57:29.784793 7886 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:57:29.774620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.2:54307 (local address 127.25.94.62:44503)
0807 22:57:29.774846 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:29.774856 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:29.775325 (+ 469us) server_negotiation.cc:407] Connection header received
0807 22:57:29.775520 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:29.775526 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:29.775628 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:29.775751 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:29.776719 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:29.777738 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:29.779223 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:29.779877 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:29.780017 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:29.784552 (+ 4535us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.94.2:54307: BlockingRecv error: recv got EOF from 127.25.94.2:54307 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Aug 07 22:57:30 dist-test-slave-z8x9 krb5kdc[6653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.3@KRBTEST.COM for oryx/127.25.94.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 22:57:30 dist-test-slave-z8x9 krb5kdc[6653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.3@KRBTEST.COM for oryx/127.25.94.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 22:57:30.265391 8390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:30.256842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:44503 (local address 127.25.94.3:50183)
0807 22:57:30.257460 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:30.257478 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 22:57:30.257592 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:57:30.257912 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:57:30.257915 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:57:30.258330 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:57:30.258495 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:30.258504 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:30.259507 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:30.259512 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:30.260430 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:30.260443 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:30.261151 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:30.261156 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:30.261442 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:30.262106 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:57:30.262125 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:57:30.265091 (+ 2966us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.94.62:44503: Server oryx/127.25.94.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":360,"threads_started":1}
W20260807 22:57:30.265595 7886 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:57:30.256959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.3:50183 (local address 127.25.94.62:44503)
0807 22:57:30.257135 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:30.257140 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:30.257570 (+ 430us) server_negotiation.cc:407] Connection header received
0807 22:57:30.257692 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:30.257696 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:30.257774 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:30.257867 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:30.258613 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:30.259371 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:30.260564 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:30.261008 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:30.261105 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:30.265343 (+ 4238us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.94.3:50183: BlockingRecv error: recv got EOF from 127.25.94.3:50183 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Aug 07 22:57:30 dist-test-slave-z8x9 krb5kdc[6653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.1@KRBTEST.COM for oryx/127.25.94.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 22:57:30 dist-test-slave-z8x9 krb5kdc[6653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.1@KRBTEST.COM for oryx/127.25.94.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 22:57:30.326539 8391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:30.317697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:44503 (local address 127.25.94.1:44763)
0807 22:57:30.318425 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:30.318452 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0807 22:57:30.318593 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:57:30.318919 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:57:30.318922 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:57:30.319448 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:57:30.319628 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:30.319638 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:30.320804 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:30.320808 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:30.321817 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:30.321829 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:30.322574 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:30.322578 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:30.322692 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:30.323325 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:57:30.323342 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:57:30.326239 (+ 2897us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.94.62:44503: Server oryx/127.25.94.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":604,"thread_start_us":350,"threads_started":1}
W20260807 22:57:30.326859 7886 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:57:30.317814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.1:44763 (local address 127.25.94.62:44503)
0807 22:57:30.318085 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:30.318094 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:30.318599 (+ 505us) server_negotiation.cc:407] Connection header received
0807 22:57:30.318702 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:30.318708 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:30.318783 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:30.318876 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:30.319785 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:30.320646 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:30.321968 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:30.322395 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:30.322514 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:30.326596 (+ 4082us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.94.1:44763: BlockingRecv error: recv got EOF from 127.25.94.1:44763 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Aug 07 22:57:30 dist-test-slave-z8x9 krb5kdc[6653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.2@KRBTEST.COM for oryx/127.25.94.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 22:57:30 dist-test-slave-z8x9 krb5kdc[6653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.2@KRBTEST.COM for oryx/127.25.94.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 22:57:30.794219 8392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:30.785414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:44503 (local address 127.25.94.2:33821)
0807 22:57:30.786181 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:30.786199 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 22:57:30.786421 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:57:30.786778 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:57:30.786783 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:57:30.787246 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:57:30.787418 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:30.787426 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:30.788484 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:30.788490 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:30.789472 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:30.789484 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:30.790165 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:30.790171 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:30.790339 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:30.790974 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:57:30.790992 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:57:30.793959 (+ 2967us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.94.62:44503: Server oryx/127.25.94.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":659,"thread_start_us":277,"threads_started":1}
W20260807 22:57:30.794425 7886 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:57:30.785452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.2:33821 (local address 127.25.94.62:44503)
0807 22:57:30.785706 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:30.785714 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:30.786314 (+ 600us) server_negotiation.cc:407] Connection header received
0807 22:57:30.786550 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:30.786556 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:30.786641 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:30.786744 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:30.787564 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:30.788343 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:30.789610 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:30.790017 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:30.790151 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:30.794156 (+ 4005us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.94.2:33821: BlockingRecv error: recv got EOF from 127.25.94.2:33821 (error 108)
Metrics: {"server-negotiator.queue_time_us":129}
Aug 07 22:57:31 dist-test-slave-z8x9 krb5kdc[6653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.3@KRBTEST.COM for oryx/127.25.94.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 22:57:31 dist-test-slave-z8x9 krb5kdc[6653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.3@KRBTEST.COM for oryx/127.25.94.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 22:57:31.275275 8394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:31.266436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:44503 (local address 127.25.94.3:60169)
0807 22:57:31.267074 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:31.267092 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 22:57:31.267212 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:57:31.267521 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:57:31.267524 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:57:31.267944 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:57:31.268110 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:31.268119 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:31.269258 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:31.269262 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:31.270449 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:31.270462 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:31.271284 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:31.271290 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:31.271470 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:31.272056 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:57:31.272073 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:57:31.275011 (+ 2938us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.94.62:44503: Server oryx/127.25.94.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":322,"threads_started":1}
W20260807 22:57:31.275398 7886 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:57:31.266549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.3:60169 (local address 127.25.94.62:44503)
0807 22:57:31.266728 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:31.266732 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:31.267195 (+ 463us) server_negotiation.cc:407] Connection header received
0807 22:57:31.267318 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:31.267322 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:31.267389 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:31.267479 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:31.268270 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:31.269037 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:31.270595 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:31.271122 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:31.271255 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:31.275203 (+ 3948us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.94.3:60169: BlockingRecv error: recv got EOF from 127.25.94.3:60169 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Aug 07 22:57:31 dist-test-slave-z8x9 krb5kdc[6653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.1@KRBTEST.COM for oryx/127.25.94.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 22:57:31 dist-test-slave-z8x9 krb5kdc[6653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.1@KRBTEST.COM for oryx/127.25.94.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 22:57:31.336421 8395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:31.327521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:44503 (local address 127.25.94.1:59753)
0807 22:57:31.328055 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:31.328075 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 22:57:31.328189 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:57:31.328487 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:57:31.328490 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:57:31.328878 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:57:31.329036 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:31.329044 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:31.330132 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:31.330136 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:31.331061 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:31.331074 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:31.332010 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:31.332016 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:31.332201 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:31.332886 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:57:31.332904 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:57:31.336146 (+ 3242us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.94.62:44503: Server oryx/127.25.94.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":286,"threads_started":1}
W20260807 22:57:31.336768 7886 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:57:31.327726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.1:59753 (local address 127.25.94.62:44503)
0807 22:57:31.327910 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:31.327915 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:31.328146 (+ 231us) server_negotiation.cc:407] Connection header received
0807 22:57:31.328287 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:31.328296 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:31.328359 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:31.328444 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:31.329207 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:31.329988 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:31.331251 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:31.331811 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:31.331913 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:31.336389 (+ 4476us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.94.1:59753: BlockingRecv error: recv got EOF from 127.25.94.1:59753 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Aug 07 22:57:31 dist-test-slave-z8x9 krb5kdc[6653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.2@KRBTEST.COM for oryx/127.25.94.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 22:57:31 dist-test-slave-z8x9 krb5kdc[6653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.2@KRBTEST.COM for oryx/127.25.94.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 22:57:31.816686 8396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:31.795466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:44503 (local address 127.25.94.2:54851)
0807 22:57:31.795998 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:31.796015 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 22:57:31.796195 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:57:31.796500 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:57:31.796504 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:57:31.796917 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:57:31.797077 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:31.797086 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:31.798640 (+ 1554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:31.798644 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:31.799762 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:31.799775 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:31.800447 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:31.800452 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:31.800575 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:31.813218 (+ 12643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:57:31.813256 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:57:31.816454 (+ 3198us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.94.62:44503: Server oryx/127.25.94.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":294,"threads_started":1}
W20260807 22:57:31.816946 7886 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:57:31.795562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.2:54851 (local address 127.25.94.62:44503)
0807 22:57:31.795792 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:31.795799 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:31.796104 (+ 305us) server_negotiation.cc:407] Connection header received
0807 22:57:31.796292 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:31.796297 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:31.796370 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:31.796451 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:31.797245 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:31.798490 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:31.799893 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:31.800288 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:31.800389 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:31.816663 (+ 16274us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.94.2:54851: BlockingRecv error: recv got EOF from 127.25.94.2:54851 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Aug 07 22:57:32 dist-test-slave-z8x9 krb5kdc[6653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.3@KRBTEST.COM for oryx/127.25.94.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 22:57:32 dist-test-slave-z8x9 krb5kdc[6653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.3@KRBTEST.COM for oryx/127.25.94.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 22:57:32.285429 8397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:32.276484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:44503 (local address 127.25.94.3:45029)
0807 22:57:32.277095 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:32.277118 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0807 22:57:32.277302 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:57:32.277734 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:57:32.277740 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:57:32.278298 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:57:32.278465 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:32.278474 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:32.279473 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:32.279476 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:32.280575 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:32.280588 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:32.281350 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:32.281355 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:32.281492 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:32.282168 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:57:32.282186 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:57:32.285034 (+ 2848us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.94.62:44503: Server oryx/127.25.94.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":266,"threads_started":1}
W20260807 22:57:32.285560 7886 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:57:32.276550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.3:45029 (local address 127.25.94.62:44503)
0807 22:57:32.276834 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:32.276840 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:32.277277 (+ 437us) server_negotiation.cc:407] Connection header received
0807 22:57:32.277462 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:32.277468 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:32.277552 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:32.277659 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:32.278597 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:32.279341 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:32.280719 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:32.281186 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:32.281291 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:32.285282 (+ 3991us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.94.3:45029: BlockingRecv error: recv got EOF from 127.25.94.3:45029 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Aug 07 22:57:32 dist-test-slave-z8x9 krb5kdc[6653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.1@KRBTEST.COM for oryx/127.25.94.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 22:57:32 dist-test-slave-z8x9 krb5kdc[6653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.1@KRBTEST.COM for oryx/127.25.94.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 22:57:32.346995 8401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:32.337592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:44503 (local address 127.25.94.1:59253)
0807 22:57:32.338250 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:32.338277 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0807 22:57:32.338424 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:57:32.338864 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:57:32.338869 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:57:32.339433 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:57:32.339616 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:32.339626 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:32.340788 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:32.340792 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:32.341975 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:32.341988 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:32.342757 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:32.342762 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:32.342890 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:32.343606 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:57:32.343625 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:57:32.346750 (+ 3125us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.94.62:44503: Server oryx/127.25.94.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":329,"threads_started":1}
W20260807 22:57:32.347182 7886 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:57:32.337671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.1:59253 (local address 127.25.94.62:44503)
0807 22:57:32.337921 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:32.337929 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:32.338526 (+ 597us) server_negotiation.cc:407] Connection header received
0807 22:57:32.338596 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:32.338601 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:32.338689 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:32.338791 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:32.339784 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:32.340622 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:32.342114 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:32.342600 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:32.342734 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:32.346968 (+ 4234us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.94.1:59253: BlockingRecv error: recv got EOF from 127.25.94.1:59253 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Aug 07 22:57:32 dist-test-slave-z8x9 krb5kdc[6653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.2@KRBTEST.COM for oryx/127.25.94.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 22:57:32 dist-test-slave-z8x9 krb5kdc[6653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.2@KRBTEST.COM for oryx/127.25.94.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 22:57:32.826630 8402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:32.817819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:44503 (local address 127.25.94.2:55425)
0807 22:57:32.818422 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:32.818438 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 22:57:32.818551 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:57:32.818831 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:57:32.818834 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:57:32.819335 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:57:32.819560 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:32.819570 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:32.820549 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:32.820552 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:32.821543 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:32.821555 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:32.822302 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:32.822307 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:32.822442 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:32.823075 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:57:32.823094 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:57:32.826363 (+ 3269us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.94.62:44503: Server oryx/127.25.94.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":347,"threads_started":1}
W20260807 22:57:32.826982 7886 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:57:32.817848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.2:55425 (local address 127.25.94.62:44503)
0807 22:57:32.818018 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:32.818023 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:32.818508 (+ 485us) server_negotiation.cc:407] Connection header received
0807 22:57:32.818645 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:32.818649 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:32.818710 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:32.818792 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:32.819683 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:32.820429 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:32.821669 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:32.822106 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:32.822207 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:32.826676 (+ 4469us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.94.2:55425: BlockingRecv error: recv got EOF from 127.25.94.2:55425 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Aug 07 22:57:33 dist-test-slave-z8x9 krb5kdc[6653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.3@KRBTEST.COM for oryx/127.25.94.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 22:57:33 dist-test-slave-z8x9 krb5kdc[6653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.3@KRBTEST.COM for oryx/127.25.94.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 22:57:33.295600 8403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:33.286606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:44503 (local address 127.25.94.3:47089)
0807 22:57:33.287199 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:33.287231 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0807 22:57:33.287389 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:57:33.287765 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:57:33.287768 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:57:33.288169 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:57:33.288380 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:33.288390 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:33.289612 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:33.289617 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:33.290630 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:33.290644 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:33.291385 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:33.291390 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:33.291513 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:33.292110 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:57:33.292128 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:57:33.295334 (+ 3206us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.94.62:44503: Server oryx/127.25.94.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":305,"threads_started":1}
W20260807 22:57:33.295858 7886 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:57:33.286634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.3:47089 (local address 127.25.94.62:44503)
0807 22:57:33.286890 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:33.286898 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:33.287481 (+ 583us) server_negotiation.cc:407] Connection header received
0807 22:57:33.287550 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:33.287557 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:33.287639 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:33.287715 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:33.288565 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:33.289447 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:33.290791 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:33.291225 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:33.291346 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:33.295596 (+ 4250us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.94.3:47089: BlockingRecv error: recv got EOF from 127.25.94.3:47089 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Aug 07 22:57:33 dist-test-slave-z8x9 krb5kdc[6653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.1@KRBTEST.COM for oryx/127.25.94.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 22:57:33 dist-test-slave-z8x9 krb5kdc[6653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.1@KRBTEST.COM for oryx/127.25.94.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 22:57:33.356539 8406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:33.348016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:44503 (local address 127.25.94.1:44131)
0807 22:57:33.348589 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:33.348608 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 22:57:33.348720 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:57:33.349282 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:57:33.349286 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:57:33.349690 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:57:33.349840 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:33.349849 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:33.350860 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:33.350864 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:33.351839 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:33.351851 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:33.352568 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:33.352573 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:33.352695 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:33.353343 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:57:33.353361 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:57:33.356299 (+ 2938us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.94.62:44503: Server oryx/127.25.94.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":376,"threads_started":1}
W20260807 22:57:33.356812 7886 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:57:33.348082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.1:44131 (local address 127.25.94.62:44503)
0807 22:57:33.348267 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:33.348271 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:33.348694 (+ 423us) server_negotiation.cc:407] Connection header received
0807 22:57:33.348997 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:33.349001 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:33.349066 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:33.349147 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:33.349976 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:33.350735 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:33.351973 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:33.352421 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:33.352519 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:33.356530 (+ 4011us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.94.1:44131: BlockingRecv error: recv got EOF from 127.25.94.1:44131 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Aug 07 22:57:33 dist-test-slave-z8x9 krb5kdc[6653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.2@KRBTEST.COM for oryx/127.25.94.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 22:57:33 dist-test-slave-z8x9 krb5kdc[6653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.2@KRBTEST.COM for oryx/127.25.94.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 22:57:33.836299 8407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:33.827808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:44503 (local address 127.25.94.2:57341)
0807 22:57:33.828356 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:33.828373 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 22:57:33.828524 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:57:33.828859 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:57:33.828864 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:57:33.829371 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:57:33.829520 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:33.829530 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:33.830571 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:33.830581 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:33.831497 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:33.831509 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:33.832219 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:33.832225 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:33.832418 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:33.833065 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:57:33.833083 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:57:33.835949 (+ 2866us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.94.62:44503: Server oryx/127.25.94.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":334,"threads_started":1}
W20260807 22:57:33.836414 7886 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:57:33.827882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.2:57341 (local address 127.25.94.62:44503)
0807 22:57:33.828130 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:33.828139 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:33.828456 (+ 317us) server_negotiation.cc:407] Connection header received
0807 22:57:33.828646 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:33.828656 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:33.828719 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:33.828832 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:33.829670 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:33.830425 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:33.831636 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:33.832064 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:33.832201 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:33.836160 (+ 3959us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.94.2:57341: BlockingRecv error: recv got EOF from 127.25.94.2:57341 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Aug 07 22:57:34 dist-test-slave-z8x9 krb5kdc[6653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.3@KRBTEST.COM for oryx/127.25.94.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 22:57:34 dist-test-slave-z8x9 krb5kdc[6653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.3@KRBTEST.COM for oryx/127.25.94.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 22:57:34.305296 8408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:34.296693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:44503 (local address 127.25.94.3:55493)
0807 22:57:34.297314 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:34.297332 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 22:57:34.297445 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:57:34.297738 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:57:34.297741 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:57:34.298135 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:57:34.298308 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:34.298317 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:34.299302 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:34.299306 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:34.300219 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:34.300232 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:34.301131 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:34.301137 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:34.301399 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:34.302116 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:57:34.302135 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:57:34.304992 (+ 2857us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.94.62:44503: Server oryx/127.25.94.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":338,"threads_started":1}
W20260807 22:57:34.305562 7886 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:57:34.296919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.3:55493 (local address 127.25.94.62:44503)
0807 22:57:34.297154 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:34.297160 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:34.297410 (+ 250us) server_negotiation.cc:407] Connection header received
0807 22:57:34.297545 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:34.297549 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:34.297611 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:34.297705 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:34.298443 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:34.299163 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:34.300372 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:34.300947 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:34.301085 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:34.305297 (+ 4212us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.94.3:55493: BlockingRecv error: recv got EOF from 127.25.94.3:55493 (error 108)
Metrics: {"server-negotiator.queue_time_us":107}
Aug 07 22:57:34 dist-test-slave-z8x9 krb5kdc[6653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.1@KRBTEST.COM for oryx/127.25.94.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 22:57:34 dist-test-slave-z8x9 krb5kdc[6653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.1@KRBTEST.COM for oryx/127.25.94.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 22:57:34.367398 8409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:34.357790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:44503 (local address 127.25.94.1:52243)
0807 22:57:34.358325 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:34.358347 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 22:57:34.358477 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:57:34.358886 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:57:34.358889 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:57:34.359370 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:57:34.359548 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:34.359558 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:34.360670 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:34.360676 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:34.361756 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:34.361771 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:34.362724 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:34.362735 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:34.362970 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:34.363820 (+ 850us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:57:34.363839 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:57:34.367035 (+ 3196us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.94.62:44503: Server oryx/127.25.94.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":282,"threads_started":1}
W20260807 22:57:34.367539 7886 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:57:34.357897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.1:52243 (local address 127.25.94.62:44503)
0807 22:57:34.358236 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:34.358242 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:34.358436 (+ 194us) server_negotiation.cc:407] Connection header received
0807 22:57:34.358638 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:34.358643 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:34.358728 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:34.358821 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:34.359735 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:34.360502 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:34.361938 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:34.362510 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:34.362629 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:34.367266 (+ 4637us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.94.1:52243: BlockingRecv error: recv got EOF from 127.25.94.1:52243 (error 108)
Metrics: {"server-negotiator.queue_time_us":110}
Aug 07 22:57:34 dist-test-slave-z8x9 krb5kdc[6653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.2@KRBTEST.COM for oryx/127.25.94.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 22:57:34 dist-test-slave-z8x9 krb5kdc[6653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.2@KRBTEST.COM for oryx/127.25.94.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 22:57:34.846071 8411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:34.837432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:44503 (local address 127.25.94.2:56387)
0807 22:57:34.837996 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:34.838016 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 22:57:34.838142 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:57:34.838483 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:57:34.838486 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:57:34.838907 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:57:34.839102 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:34.839111 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:34.840318 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:34.840325 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:34.841296 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:34.841309 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:34.841997 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:34.842004 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:34.842188 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:34.842837 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:57:34.842856 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:57:34.845824 (+ 2968us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.94.62:44503: Server oryx/127.25.94.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":336,"threads_started":1}
W20260807 22:57:34.846328 7886 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:57:34.837505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.2:56387 (local address 127.25.94.62:44503)
0807 22:57:34.837710 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:34.837716 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:34.838121 (+ 405us) server_negotiation.cc:407] Connection header received
0807 22:57:34.838266 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:34.838270 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:34.838337 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:34.838439 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:34.839279 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:34.840160 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:34.841431 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:34.841837 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:34.841968 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:34.846049 (+ 4081us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.94.2:56387: BlockingRecv error: recv got EOF from 127.25.94.2:56387 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Aug 07 22:57:35 dist-test-slave-z8x9 krb5kdc[6653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.3@KRBTEST.COM for oryx/127.25.94.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 22:57:35 dist-test-slave-z8x9 krb5kdc[6653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.3@KRBTEST.COM for oryx/127.25.94.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 22:57:35.315907 8412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:35.306433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:44503 (local address 127.25.94.3:36957)
0807 22:57:35.307099 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:35.307121 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 22:57:35.307267 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:57:35.307675 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:57:35.307679 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:57:35.308122 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:57:35.308355 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:35.308365 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:35.309549 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:35.309553 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:35.310486 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:35.310499 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:35.311334 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:35.311340 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:35.311465 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:35.312336 (+ 871us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:57:35.312357 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:57:35.315576 (+ 3219us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.94.62:44503: Server oryx/127.25.94.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":376,"threads_started":1}
W20260807 22:57:35.316159 7886 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:57:35.306764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.3:36957 (local address 127.25.94.62:44503)
0807 22:57:35.306945 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:35.306950 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:35.307254 (+ 304us) server_negotiation.cc:407] Connection header received
0807 22:57:35.307400 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:35.307405 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:35.307492 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:35.307591 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:35.308548 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:35.309381 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:35.310644 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:35.311095 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:35.311204 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:35.315874 (+ 4670us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.94.3:36957: BlockingRecv error: recv got EOF from 127.25.94.3:36957 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Aug 07 22:57:35 dist-test-slave-z8x9 krb5kdc[6653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.1@KRBTEST.COM for oryx/127.25.94.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 22:57:35 dist-test-slave-z8x9 krb5kdc[6653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.1@KRBTEST.COM for oryx/127.25.94.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 22:57:35.377185 8413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:35.368395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:44503 (local address 127.25.94.1:54135)
0807 22:57:35.368936 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:35.368957 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 22:57:35.369073 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:57:35.369392 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:57:35.369396 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:57:35.369826 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:57:35.369984 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:35.369992 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:35.370987 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:35.370991 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:35.371944 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:35.371956 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:35.372688 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:35.372694 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:35.372908 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:35.373666 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:57:35.373684 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:57:35.376779 (+ 3095us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.94.62:44503: Server oryx/127.25.94.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":312,"threads_started":1}
W20260807 22:57:35.377334 7886 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:57:35.368521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.1:54135 (local address 127.25.94.62:44503)
0807 22:57:35.368819 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:35.368826 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:35.369036 (+ 210us) server_negotiation.cc:407] Connection header received
0807 22:57:35.369192 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:35.369197 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:35.369261 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:35.369350 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:35.370112 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:35.370846 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:35.372061 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:35.372529 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:35.372677 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:35.377020 (+ 4343us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.94.1:54135: BlockingRecv error: recv got EOF from 127.25.94.1:54135 (error 108)
Metrics: {"server-negotiator.queue_time_us":119}
Aug 07 22:57:35 dist-test-slave-z8x9 krb5kdc[6653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.2@KRBTEST.COM for oryx/127.25.94.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 22:57:35 dist-test-slave-z8x9 krb5kdc[6653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.2@KRBTEST.COM for oryx/127.25.94.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 22:57:35.855949 8415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:35.847167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:44503 (local address 127.25.94.2:43123)
0807 22:57:35.847805 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:35.847829 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0807 22:57:35.847983 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:57:35.848324 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:57:35.848327 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:57:35.848735 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:57:35.848899 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:35.848909 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:35.850020 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:35.850025 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:35.850996 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:35.851010 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:35.851825 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:35.851830 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:35.851978 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:35.852710 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:57:35.852729 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:57:35.855602 (+ 2873us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.94.62:44503: Server oryx/127.25.94.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":329,"threads_started":1}
W20260807 22:57:35.856055 7886 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:57:35.847263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.2:43123 (local address 127.25.94.62:44503)
0807 22:57:35.847490 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:35.847495 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:35.847919 (+ 424us) server_negotiation.cc:407] Connection header received
0807 22:57:35.848104 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:35.848109 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:35.848173 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:35.848253 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:35.849064 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:35.849857 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:35.851152 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:35.851664 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:35.851765 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:35.855823 (+ 4058us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.94.2:43123: BlockingRecv error: recv got EOF from 127.25.94.2:43123 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Aug 07 22:57:36 dist-test-slave-z8x9 krb5kdc[6653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.3@KRBTEST.COM for oryx/127.25.94.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 22:57:36 dist-test-slave-z8x9 krb5kdc[6653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.3@KRBTEST.COM for oryx/127.25.94.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 22:57:36.326129 8416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:36.316978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:44503 (local address 127.25.94.3:44669)
0807 22:57:36.317630 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:36.317648 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 22:57:36.317763 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:57:36.318064 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:57:36.318068 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:57:36.318530 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:57:36.318737 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:36.318747 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:36.319868 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:36.319872 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:36.320927 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:36.320940 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:36.321899 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:36.321907 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:36.322101 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:36.322841 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:57:36.322859 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:57:36.325887 (+ 3028us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.94.62:44503: Server oryx/127.25.94.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":377,"threads_started":1}
W20260807 22:57:36.326460 7886 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:57:36.317002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.3:44669 (local address 127.25.94.62:44503)
0807 22:57:36.317247 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:36.317253 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:36.317723 (+ 470us) server_negotiation.cc:407] Connection header received
0807 22:57:36.317863 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:36.317867 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:36.317931 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:36.318019 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:36.318896 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:36.319737 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:36.321102 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:36.321741 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:36.321868 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:36.326126 (+ 4258us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.94.3:44669: BlockingRecv error: recv got EOF from 127.25.94.3:44669 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Aug 07 22:57:36 dist-test-slave-z8x9 krb5kdc[6653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.1@KRBTEST.COM for oryx/127.25.94.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 22:57:36 dist-test-slave-z8x9 krb5kdc[6653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.1@KRBTEST.COM for oryx/127.25.94.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 22:57:36.386694 8417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:36.378207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:44503 (local address 127.25.94.1:51887)
0807 22:57:36.378756 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:36.378776 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 22:57:36.378951 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:57:36.379280 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:57:36.379284 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:57:36.379713 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:57:36.379874 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:36.379883 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:36.380942 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:36.380948 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:36.381907 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:36.381918 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:36.382690 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:36.382697 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:36.382866 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:36.383508 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:57:36.383526 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:57:36.386337 (+ 2811us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.94.62:44503: Server oryx/127.25.94.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":280,"threads_started":1}
W20260807 22:57:36.386811 7886 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:57:36.378360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.1:51887 (local address 127.25.94.62:44503)
0807 22:57:36.378609 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:36.378616 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:36.378869 (+ 253us) server_negotiation.cc:407] Connection header received
0807 22:57:36.379085 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:36.379089 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:36.379151 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:36.379252 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:36.380033 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:36.380798 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:36.382056 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:36.382513 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:36.382652 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:36.386561 (+ 3909us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.94.1:51887: BlockingRecv error: recv got EOF from 127.25.94.1:51887 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Aug 07 22:57:36 dist-test-slave-z8x9 krb5kdc[6653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.2@KRBTEST.COM for oryx/127.25.94.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 22:57:36 dist-test-slave-z8x9 krb5kdc[6653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.2@KRBTEST.COM for oryx/127.25.94.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 22:57:36.869144 8418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:36.857195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:44503 (local address 127.25.94.2:43739)
0807 22:57:36.857965 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:36.857986 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 22:57:36.858133 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:57:36.858484 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:57:36.858489 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:57:36.858990 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:57:36.859152 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:36.859163 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:36.860268 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:36.860274 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:36.861234 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:36.861248 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:36.862042 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:36.862048 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:36.862222 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:36.865729 (+ 3507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:57:36.865747 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:57:36.868773 (+ 3026us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.94.62:44503: Server oryx/127.25.94.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":643,"thread_start_us":299,"threads_started":1}
W20260807 22:57:36.869313 7886 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:57:36.857333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.2:43739 (local address 127.25.94.62:44503)
0807 22:57:36.857528 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:36.857535 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:36.858072 (+ 537us) server_negotiation.cc:407] Connection header received
0807 22:57:36.858263 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:36.858269 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:36.858344 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:36.858447 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:36.859301 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:36.860116 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:36.861380 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:36.861853 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:36.861983 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:36.869009 (+ 7026us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.94.2:43739: BlockingRecv error: recv got EOF from 127.25.94.2:43739 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Aug 07 22:57:37 dist-test-slave-z8x9 krb5kdc[6653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.3@KRBTEST.COM for oryx/127.25.94.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 22:57:37 dist-test-slave-z8x9 krb5kdc[6653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.3@KRBTEST.COM for oryx/127.25.94.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 22:57:37.336434 8421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:37.327285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:44503 (local address 127.25.94.3:60441)
0807 22:57:37.327900 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:37.327925 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0807 22:57:37.328069 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:57:37.328459 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:57:37.328464 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:57:37.329061 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:57:37.329273 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:37.329283 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:37.330285 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:37.330306 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:37.331231 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:37.331243 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:37.332103 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:37.332108 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:37.332291 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:37.332994 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:57:37.333014 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:57:37.336069 (+ 3055us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.94.62:44503: Server oryx/127.25.94.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":317,"threads_started":1}
W20260807 22:57:37.336607 7886 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:57:37.327403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.3:60441 (local address 127.25.94.62:44503)
0807 22:57:37.327602 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:37.327608 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:37.328013 (+ 405us) server_negotiation.cc:407] Connection header received
0807 22:57:37.328202 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:37.328208 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:37.328296 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:37.328403 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:37.329401 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:37.330138 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:37.331431 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:37.331910 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:37.332019 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:37.336341 (+ 4322us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.94.3:60441: BlockingRecv error: recv got EOF from 127.25.94.3:60441 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Aug 07 22:57:37 dist-test-slave-z8x9 krb5kdc[6653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.1@KRBTEST.COM for oryx/127.25.94.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 22:57:37 dist-test-slave-z8x9 krb5kdc[6653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.1@KRBTEST.COM for oryx/127.25.94.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 22:57:37.396922 8424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:37.387847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:44503 (local address 127.25.94.1:48529)
0807 22:57:37.388429 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:37.388457 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0807 22:57:37.388619 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:57:37.389082 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:57:37.389087 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:57:37.389678 (+ 591us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:57:37.389888 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:37.389899 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:37.391135 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:37.391140 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:37.392068 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:37.392079 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:37.392799 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:37.392805 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:37.392986 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:37.393712 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:57:37.393730 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:57:37.396532 (+ 2802us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.94.62:44503: Server oryx/127.25.94.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":268,"threads_started":1}
W20260807 22:57:37.397032 7886 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:57:37.387959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.1:48529 (local address 127.25.94.62:44503)
0807 22:57:37.388379 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:37.388386 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:37.388544 (+ 158us) server_negotiation.cc:407] Connection header received
0807 22:57:37.388755 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:37.388760 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:37.388852 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:37.388961 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:37.390047 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:37.390971 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:37.392219 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:37.392644 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:37.392776 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:37.396756 (+ 3980us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.94.1:48529: BlockingRecv error: recv got EOF from 127.25.94.1:48529 (error 108)
Metrics: {"server-negotiator.queue_time_us":237}
Aug 07 22:57:37 dist-test-slave-z8x9 krb5kdc[6653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.2@KRBTEST.COM for oryx/127.25.94.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 22:57:37 dist-test-slave-z8x9 krb5kdc[6653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.2@KRBTEST.COM for oryx/127.25.94.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 22:57:37.878831 8425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:37.870311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:44503 (local address 127.25.94.2:57883)
0807 22:57:37.870816 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:37.870833 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 22:57:37.870996 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:57:37.871355 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:57:37.871360 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:57:37.871788 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:57:37.871971 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:37.871980 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:37.873024 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:37.873030 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:37.873973 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:37.873986 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:37.874667 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:37.874673 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:37.874848 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:37.875486 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:57:37.875504 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:57:37.878489 (+ 2985us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.94.62:44503: Server oryx/127.25.94.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":272,"threads_started":1}
W20260807 22:57:37.878979 7886 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:57:37.870423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.2:57883 (local address 127.25.94.62:44503)
0807 22:57:37.870664 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:37.870671 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:37.870924 (+ 253us) server_negotiation.cc:407] Connection header received
0807 22:57:37.871129 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:37.871139 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:37.871213 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:37.871319 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:37.872122 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:37.872876 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:37.874116 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:37.874515 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:37.874642 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:37.878697 (+ 4055us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.94.2:57883: BlockingRecv error: recv got EOF from 127.25.94.2:57883 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Aug 07 22:57:38 dist-test-slave-z8x9 krb5kdc[6653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.3@KRBTEST.COM for oryx/127.25.94.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 22:57:38 dist-test-slave-z8x9 krb5kdc[6653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.3@KRBTEST.COM for oryx/127.25.94.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 22:57:38.347213 8426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:38.337581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:44503 (local address 127.25.94.3:50437)
0807 22:57:38.338140 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:38.338160 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 22:57:38.338280 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:57:38.338689 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:57:38.338698 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:57:38.339285 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:57:38.339516 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:38.339528 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:38.341311 (+ 1783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:38.341318 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:38.342445 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:38.342459 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:38.343154 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:38.343158 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:38.343354 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:38.344030 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:57:38.344048 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:57:38.346953 (+ 2905us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.94.62:44503: Server oryx/127.25.94.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":321,"threads_started":1}
W20260807 22:57:38.347429 7886 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:57:38.337765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.3:50437 (local address 127.25.94.62:44503)
0807 22:57:38.337948 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:38.337953 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:38.338236 (+ 283us) server_negotiation.cc:407] Connection header received
0807 22:57:38.338384 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:38.338389 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:38.338477 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:38.338572 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:38.339834 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:38.341043 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:38.342598 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:38.343028 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:38.343131 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:38.347169 (+ 4038us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.94.3:50437: BlockingRecv error: recv got EOF from 127.25.94.3:50437 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Aug 07 22:57:38 dist-test-slave-z8x9 krb5kdc[6653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.1@KRBTEST.COM for oryx/127.25.94.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 22:57:38 dist-test-slave-z8x9 krb5kdc[6653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.1@KRBTEST.COM for oryx/127.25.94.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 22:57:38.407171 8429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:38.398089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:44503 (local address 127.25.94.1:57151)
0807 22:57:38.398596 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:38.398616 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 22:57:38.398733 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:57:38.399036 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:57:38.399039 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:57:38.399457 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:57:38.399646 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:38.399654 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:38.400922 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:38.400930 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:38.402473 (+ 1543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:38.402487 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:38.403235 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:38.403240 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:38.403371 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:38.403994 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:57:38.404012 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:57:38.406898 (+ 2886us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.94.62:44503: Server oryx/127.25.94.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":295,"threads_started":1}
W20260807 22:57:38.407428 7886 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:57:38.398274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.1:57151 (local address 127.25.94.62:44503)
0807 22:57:38.398523 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:38.398529 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:38.398689 (+ 160us) server_negotiation.cc:407] Connection header received
0807 22:57:38.398853 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:38.398857 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:38.398920 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:38.398996 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:38.399851 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:38.400725 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:38.402623 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:38.403071 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:38.403178 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:38.407159 (+ 3981us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.94.1:57151: BlockingRecv error: recv got EOF from 127.25.94.1:57151 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Aug 07 22:57:38 dist-test-slave-z8x9 krb5kdc[6653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.2@KRBTEST.COM for oryx/127.25.94.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 22:57:38 dist-test-slave-z8x9 krb5kdc[6653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.94.2@KRBTEST.COM for oryx/127.25.94.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 22:57:38.888963 8430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:38.880218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:44503 (local address 127.25.94.2:48029)
0807 22:57:38.880843 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:38.880860 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 22:57:38.880971 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:57:38.881310 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:57:38.881314 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:57:38.881763 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:57:38.881965 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:38.881974 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:38.882992 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:38.882996 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:38.883900 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:38.883912 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:38.884586 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:38.884590 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:38.884708 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:38.885382 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:57:38.885400 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:57:38.888703 (+ 3303us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.94.62:44503: Server oryx/127.25.94.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":385,"threads_started":1}
W20260807 22:57:38.889122 7886 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:57:38.880322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.2:48029 (local address 127.25.94.62:44503)
0807 22:57:38.880528 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:38.880533 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:38.880934 (+ 401us) server_negotiation.cc:407] Connection header received
0807 22:57:38.881064 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:38.881068 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:38.881128 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:38.881287 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:38.882099 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:38.882866 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:38.884015 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:38.884461 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:38.884552 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:38.888897 (+ 4345us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.94.2:48029: BlockingRecv error: recv got EOF from 127.25.94.2:48029 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
I20260807 22:57:39.177068 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 7645
I20260807 22:57:39.190475 8434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:39.178353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46388 (local address 127.25.94.1:39569)
0807 22:57:39.178903 (+ 550us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:39.178911 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:39.178940 (+ 29us) server_negotiation.cc:407] Connection header received
0807 22:57:39.179066 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:39.179073 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:39.179300 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:39.179456 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:39.180938 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:39.182050 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:39.183339 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:39.183990 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:39.184105 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:39.186983 (+ 2878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:57:39.186993 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:57:39.187009 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:57:39.187051 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:57:39.189937 (+ 2886us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.25.94.1@KRBTEST.COM not found in keytab (ticket kvno 2)
0807 22:57:39.190234 (+ 297us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46388: Request ticket server kudu/127.25.94.1@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":272,"threads_started":1}
W20260807 22:57:39.190564 8434 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46388: Request ticket server kudu/127.25.94.1@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260807 22:57:39.190832 25976 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 7645: Not authorized: Client connection negotiation failed: client connection to 127.25.94.1:39569: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.25.94.1@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260807 22:57:39.191033 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 7645
I20260807 22:57:39.215091 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 7888
I20260807 22:57:39.228543 8435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:39.216313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35912 (local address 127.25.94.2:39931)
0807 22:57:39.216969 (+ 656us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:39.216977 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:39.217009 (+ 32us) server_negotiation.cc:407] Connection header received
0807 22:57:39.217074 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:39.217081 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:39.217388 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:39.217602 (+ 214us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:39.219073 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:39.220411 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:39.221730 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:39.222293 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:39.222450 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:39.225207 (+ 2757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:57:39.225223 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:57:39.225242 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:57:39.225289 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:57:39.227827 (+ 2538us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.25.94.2@KRBTEST.COM not found in keytab (ticket kvno 2)
0807 22:57:39.228174 (+ 347us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35912: Request ticket server kudu/127.25.94.2@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":510,"thread_start_us":344,"threads_started":1}
W20260807 22:57:39.228669 8435 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35912: Request ticket server kudu/127.25.94.2@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260807 22:57:39.228641 25976 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 7888: Not authorized: Client connection negotiation failed: client connection to 127.25.94.2:39931: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.25.94.2@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260807 22:57:39.228802 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 7888
I20260807 22:57:39.253743 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 8130
I20260807 22:57:39.266770 8436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:39.254760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40918 (local address 127.25.94.3:44957)
0807 22:57:39.255399 (+ 639us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:39.255407 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:39.255433 (+ 26us) server_negotiation.cc:407] Connection header received
0807 22:57:39.255501 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:39.255516 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:39.255740 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:39.255913 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:39.257581 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:39.258706 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:39.260072 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:39.260704 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:39.260832 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:39.263644 (+ 2812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:57:39.263654 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:57:39.263669 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:57:39.263772 (+ 103us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:57:39.266244 (+ 2472us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.25.94.3@KRBTEST.COM not found in keytab (ticket kvno 2)
0807 22:57:39.266528 (+ 284us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40918: Request ticket server kudu/127.25.94.3@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":474,"thread_start_us":348,"threads_started":1}
W20260807 22:57:39.266903 8436 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40918: Request ticket server kudu/127.25.94.3@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260807 22:57:39.266994 25976 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 8130: Not authorized: Client connection negotiation failed: client connection to 127.25.94.3:44957: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.25.94.3@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260807 22:57:39.267206 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 8130
I20260807 22:57:39.291495 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 7534
I20260807 22:57:39.292984 7633 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 22:57:39.415064 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 7534
2026-08-07T22:57:39Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21784 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-07T22:57:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T22:57:39Z Disabled control of system clock
I20260807 22:57:39.509586 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786143361779964-25976-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786143361779964-25976-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786143361779964-25976-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.62:37751
--webserver_interface=127.25.94.62
--webserver_port=0
--builtin_ntp_servers=127.25.94.20:45677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.94.62:37751
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260807 22:57:39.861811 8444 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:57:39.862118 8444 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:57:39.862284 8444 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:57:39.869635 8444 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 22:57:39.869760 8444 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:57:39.869846 8444 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:57:39.869962 8444 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 22:57:39.870074 8444 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 22:57:39.879323 8444 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:45677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786143361779964-25976-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786143361779964-25976-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.94.62:37751
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.62:37751
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.94.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786143361779964-25976-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 22:57:39.880607 8444 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:57:39.882407 8444 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:57:39.895046 8458 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:57:39.895136 8454 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:57:39.895125 8455 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:57:39.895519 8444 server_base.cc:1034] running on GCE node
I20260807 22:57:39.896680 8444 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:57:39.898182 8444 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:57:39.899421 8444 hybrid_clock.cc:648] HybridClock initialized: now 1786143459899379 us; error 36 us; skew 500 ppm
I20260807 22:57:39.899791 8444 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 22:57:39.902179 8444 webserver.cc:460] Webserver started at http://127.25.94.62:43873/ using document root <none> and password file <none>
I20260807 22:57:39.902901 8444 fs_manager.cc:359] Metadata directory not provided
I20260807 22:57:39.903043 8444 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:57:39.903396 8444 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:57:39.906456 8444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786143361779964-25976-0/minicluster-data/master-0/data/instance:
uuid: "d6ffd328c3de4fe5b327de337cf4ddc6"
format_stamp: "Formatted at 2026-08-07 22:57:39 on dist-test-slave-z8x9"
server_key: "bcaefa0dafcd36c32e2e933a844b31ef"
server_key_iv: "55226a6aa9430e76faf4154c5143fac9"
server_key_version: "encryptionkey@0"
I20260807 22:57:39.907261 8444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786143361779964-25976-0/minicluster-data/master-0/wal/instance:
uuid: "d6ffd328c3de4fe5b327de337cf4ddc6"
format_stamp: "Formatted at 2026-08-07 22:57:39 on dist-test-slave-z8x9"
server_key: "bcaefa0dafcd36c32e2e933a844b31ef"
server_key_iv: "55226a6aa9430e76faf4154c5143fac9"
server_key_version: "encryptionkey@0"
I20260807 22:57:39.912545 8444 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260807 22:57:39.916297 8467 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:57:39.917979 8444 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 22:57:39.918179 8444 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786143361779964-25976-0/minicluster-data/master-0/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786143361779964-25976-0/minicluster-data/master-0/wal
uuid: "d6ffd328c3de4fe5b327de337cf4ddc6"
format_stamp: "Formatted at 2026-08-07 22:57:39 on dist-test-slave-z8x9"
server_key: "bcaefa0dafcd36c32e2e933a844b31ef"
server_key_iv: "55226a6aa9430e76faf4154c5143fac9"
server_key_version: "encryptionkey@0"
I20260807 22:57:39.918418 8444 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786143361779964-25976-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786143361779964-25976-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786143361779964-25976-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:57:39.929623 8444 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:57:39.930431 8444 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:57:39.930737 8444 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:57:39.949020 8444 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.62:37751
I20260807 22:57:39.949054 8558 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.62:37751 every 8 connection(s)
I20260807 22:57:39.950425 8444 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
I20260807 22:57:39.954141 8559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:57:39.957314 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 8444
I20260807 22:57:39.957512 25976 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786143361779964-25976-0/minicluster-data/master-0/wal/instance
I20260807 22:57:39.958160 25976 external_mini_cluster.cc:1428] Setting key 9684d02785e71ce90404b910ae611bc5
I20260807 22:57:39.967986 8559 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6ffd328c3de4fe5b327de337cf4ddc6: Bootstrap starting.
I20260807 22:57:39.972026 8559 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d6ffd328c3de4fe5b327de337cf4ddc6: Neither blocks nor log segments found. Creating new log.
I20260807 22:57:39.973456 8559 log.cc:824] T 00000000000000000000000000000000 P d6ffd328c3de4fe5b327de337cf4ddc6: Log is configured to *not* fsync() on all Append() calls
I20260807 22:57:39.974032 8565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:39.960304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44824 (local address 127.25.94.62:37751)
0807 22:57:39.961399 (+ 1095us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:39.961414 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:39.961466 (+ 52us) server_negotiation.cc:407] Connection header received
0807 22:57:39.963304 (+ 1838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:39.963349 (+ 45us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:39.964006 (+ 657us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:39.964486 (+ 480us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:39.965070 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:39.967049 (+ 1979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:39.968429 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:39.969517 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:39.969731 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:39.970922 (+ 1191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:57:39.970936 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:57:39.970957 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 22:57:39.971702 (+ 745us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:57:39.971836 (+ 134us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 22:57:39.971866 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:57:39.971992 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0807 22:57:39.972330 (+ 338us) server_negotiation.cc:299] Negotiation successful
0807 22:57:39.972697 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":581,"thread_start_us":348,"threads_started":1}
I20260807 22:57:39.976990 8559 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6ffd328c3de4fe5b327de337cf4ddc6: No bootstrap required, opened a new log
I20260807 22:57:39.984548 8559 raft_consensus.cc:348] T 00000000000000000000000000000000 P d6ffd328c3de4fe5b327de337cf4ddc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6ffd328c3de4fe5b327de337cf4ddc6" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 37751 } }
I20260807 22:57:39.984972 8559 raft_consensus.cc:374] T 00000000000000000000000000000000 P d6ffd328c3de4fe5b327de337cf4ddc6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:57:39.985091 8559 raft_consensus.cc:729] T 00000000000000000000000000000000 P d6ffd328c3de4fe5b327de337cf4ddc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6ffd328c3de4fe5b327de337cf4ddc6, State: Initialized, Role: FOLLOWER
I20260807 22:57:39.985719 8559 consensus_queue.cc:260] T 00000000000000000000000000000000 P d6ffd328c3de4fe5b327de337cf4ddc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6ffd328c3de4fe5b327de337cf4ddc6" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 37751 } }
I20260807 22:57:39.985977 8559 raft_consensus.cc:388] T 00000000000000000000000000000000 P d6ffd328c3de4fe5b327de337cf4ddc6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 22:57:39.986132 8559 raft_consensus.cc:482] T 00000000000000000000000000000000 P d6ffd328c3de4fe5b327de337cf4ddc6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 22:57:39.986318 8559 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d6ffd328c3de4fe5b327de337cf4ddc6 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:57:39.988596 8559 raft_consensus.cc:504] T 00000000000000000000000000000000 P d6ffd328c3de4fe5b327de337cf4ddc6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6ffd328c3de4fe5b327de337cf4ddc6" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 37751 } }
I20260807 22:57:39.989142 8559 leader_election.cc:302] T 00000000000000000000000000000000 P d6ffd328c3de4fe5b327de337cf4ddc6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d6ffd328c3de4fe5b327de337cf4ddc6; no voters:
I20260807 22:57:39.989828 8559 leader_election.cc:288] T 00000000000000000000000000000000 P d6ffd328c3de4fe5b327de337cf4ddc6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 22:57:39.989981 8567 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d6ffd328c3de4fe5b327de337cf4ddc6 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:57:39.991137 8567 raft_consensus.cc:686] T 00000000000000000000000000000000 P d6ffd328c3de4fe5b327de337cf4ddc6 [term 1 LEADER]: Becoming Leader. State: Replica: d6ffd328c3de4fe5b327de337cf4ddc6, State: Running, Role: LEADER
I20260807 22:57:39.991650 8567 consensus_queue.cc:237] T 00000000000000000000000000000000 P d6ffd328c3de4fe5b327de337cf4ddc6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6ffd328c3de4fe5b327de337cf4ddc6" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 37751 } }
I20260807 22:57:39.991955 8559 sys_catalog.cc:564] T 00000000000000000000000000000000 P d6ffd328c3de4fe5b327de337cf4ddc6 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 22:57:39.994712 8570 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6ffd328c3de4fe5b327de337cf4ddc6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d6ffd328c3de4fe5b327de337cf4ddc6. Latest consensus state: current_term: 1 leader_uuid: "d6ffd328c3de4fe5b327de337cf4ddc6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6ffd328c3de4fe5b327de337cf4ddc6" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 37751 } } }
I20260807 22:57:39.995065 8570 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6ffd328c3de4fe5b327de337cf4ddc6 [sys.catalog]: This master's current role is: LEADER
I20260807 22:57:39.995582 8569 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6ffd328c3de4fe5b327de337cf4ddc6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d6ffd328c3de4fe5b327de337cf4ddc6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6ffd328c3de4fe5b327de337cf4ddc6" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 37751 } } }
I20260807 22:57:39.995908 8569 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6ffd328c3de4fe5b327de337cf4ddc6 [sys.catalog]: This master's current role is: LEADER
I20260807 22:57:39.997908 8578 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 22:57:40.004334 8578 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 22:57:40.014912 8578 catalog_manager.cc:1347] Generated new cluster ID: 91e422565ee044fdad5b7a2fd69c81b5
I20260807 22:57:40.015118 8578 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 22:57:40.027233 8578 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 22:57:40.029078 8578 catalog_manager.cc:1504] Loading token signing keys...
I20260807 22:57:40.040606 8578 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d6ffd328c3de4fe5b327de337cf4ddc6: Generated new TSK 0
I20260807 22:57:40.041815 8578 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 22:57:40.051332 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786143361779964-25976-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786143361779964-25976-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786143361779964-25976-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786143361779964-25976-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.1:0
--local_ip_for_outbound_sockets=127.25.94.1
--webserver_interface=127.25.94.1
--webserver_port=0
--tserver_master_addrs=127.25.94.62:37751
--builtin_ntp_servers=127.25.94.20:45677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260807 22:57:40.430925 8593 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:57:40.431213 8593 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:57:40.431378 8593 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:57:40.438611 8593 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:57:40.438728 8593 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:57:40.438907 8593 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.1
I20260807 22:57:40.447993 8593 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:45677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786143361779964-25976-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786143361779964-25976-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786143361779964-25976-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.94.1
--webserver_port=0
--tserver_master_addrs=127.25.94.62:37751
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.1
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786143361779964-25976-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 22:57:40.449358 8593 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:57:40.451292 8593 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:57:40.473344 8603 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:57:40.473366 8602 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:57:40.473589 8605 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:57:40.474133 8593 server_base.cc:1034] running on GCE node
I20260807 22:57:40.475026 8593 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:57:40.476254 8593 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:57:40.477499 8593 hybrid_clock.cc:648] HybridClock initialized: now 1786143460477442 us; error 50 us; skew 500 ppm
I20260807 22:57:40.477931 8593 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 22:57:40.480805 8593 webserver.cc:460] Webserver started at http://127.25.94.1:33443/ using document root <none> and password file <none>
I20260807 22:57:40.481824 8593 fs_manager.cc:359] Metadata directory not provided
I20260807 22:57:40.481967 8593 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:57:40.482353 8593 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:57:40.485580 8593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786143361779964-25976-0/minicluster-data/ts-0/data/instance:
uuid: "150d4a65437d45bd912999644739db37"
format_stamp: "Formatted at 2026-08-07 22:57:40 on dist-test-slave-z8x9"
server_key: "1320c2848613ca86a3fefb104f1dbc31"
server_key_iv: "8045a4310a39a1cb3f4dffd10d027603"
server_key_version: "encryptionkey@0"
I20260807 22:57:40.486429 8593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786143361779964-25976-0/minicluster-data/ts-0/wal/instance:
uuid: "150d4a65437d45bd912999644739db37"
format_stamp: "Formatted at 2026-08-07 22:57:40 on dist-test-slave-z8x9"
server_key: "1320c2848613ca86a3fefb104f1dbc31"
server_key_iv: "8045a4310a39a1cb3f4dffd10d027603"
server_key_version: "encryptionkey@0"
I20260807 22:57:40.492121 8593 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260807 22:57:40.496647 8618 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:57:40.498554 8593 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260807 22:57:40.498746 8593 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786143361779964-25976-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786143361779964-25976-0/minicluster-data/ts-0/wal
uuid: "150d4a65437d45bd912999644739db37"
format_stamp: "Formatted at 2026-08-07 22:57:40 on dist-test-slave-z8x9"
server_key: "1320c2848613ca86a3fefb104f1dbc31"
server_key_iv: "8045a4310a39a1cb3f4dffd10d027603"
server_key_version: "encryptionkey@0"
I20260807 22:57:40.498961 8593 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786143361779964-25976-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786143361779964-25976-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786143361779964-25976-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:57:40.512233 8593 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:57:40.513147 8593 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:57:40.513531 8593 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:57:40.514570 8593 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:57:40.516050 8593 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:57:40.516179 8593 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:57:40.516402 8593 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:57:40.516566 8593 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:57:40.551024 8593 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.1:39029
I20260807 22:57:40.551081 8790 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.1:39029 every 8 connection(s)
I20260807 22:57:40.552380 8593 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786143361779964-25976-0/minicluster-data/ts-0/data/info.pb
I20260807 22:57:40.553545 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 8593
I20260807 22:57:40.553766 25976 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786143361779964-25976-0/minicluster-data/ts-0/wal/instance
I20260807 22:57:40.554591 25976 external_mini_cluster.cc:1428] Setting key 390ae8aeac39e0ac89d4d13a6537961b
I20260807 22:57:40.557729 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786143361779964-25976-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786143361779964-25976-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786143361779964-25976-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786143361779964-25976-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.2:0
--local_ip_for_outbound_sockets=127.25.94.2
--webserver_interface=127.25.94.2
--webserver_port=0
--tserver_master_addrs=127.25.94.62:37751
--builtin_ntp_servers=127.25.94.20:45677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260807 22:57:40.565768 8794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:40.555288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.1:35013 (local address 127.25.94.62:37751)
0807 22:57:40.556114 (+ 826us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:40.556121 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:40.557211 (+ 1090us) server_negotiation.cc:407] Connection header received
0807 22:57:40.559041 (+ 1830us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:40.559046 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:40.559141 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:40.559258 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:40.560340 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:40.561084 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:40.562676 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:40.563113 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:40.563249 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:40.564817 (+ 1568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:57:40.564823 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:57:40.564826 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 22:57:40.564845 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:57:40.564885 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 22:57:40.564897 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:57:40.564998 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0807 22:57:40.565299 (+ 301us) server_negotiation.cc:299] Negotiation successful
0807 22:57:40.565401 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":692,"thread_start_us":549,"threads_started":1}
I20260807 22:57:40.566820 8796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:40.555892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:37751 (local address 127.25.94.1:35013)
0807 22:57:40.557038 (+ 1146us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:40.557074 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0807 22:57:40.558769 (+ 1695us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:57:40.559494 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:57:40.559502 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:57:40.559592 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:57:40.560163 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:40.560179 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:40.561267 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:40.561271 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:40.562509 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:40.562521 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:40.563441 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:40.563458 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:40.563690 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:40.564469 (+ 779us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 22:57:40.564472 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:57:40.564516 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 22:57:40.564594 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:57:40.564999 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:57:40.565013 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:57:40.565023 (+ 10us) client_negotiation.cc:769] Sending connection context
0807 22:57:40.565299 (+ 276us) client_negotiation.cc:240] Negotiation successful
0807 22:57:40.565694 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":621,"thread_start_us":357,"threads_started":1}
I20260807 22:57:40.568678 8792 heartbeater.cc:347] Connected to a master server at 127.25.94.62:37751
I20260807 22:57:40.569039 8792 heartbeater.cc:464] Registering TS with master...
I20260807 22:57:40.570102 8792 heartbeater.cc:511] Master 127.25.94.62:37751 requested a full tablet report, sending...
I20260807 22:57:40.572510 8495 ts_manager.cc:194] Registered new tserver with Master: 150d4a65437d45bd912999644739db37 (127.25.94.1:39029)
I20260807 22:57:40.574265 8495 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.94.1:35013
W20260807 22:57:40.899910 8797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:57:40.900177 8797 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:57:40.900336 8797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:57:40.908627 8797 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:57:40.908764 8797 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:57:40.908933 8797 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.2
I20260807 22:57:40.918431 8797 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:45677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786143361779964-25976-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786143361779964-25976-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786143361779964-25976-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.94.2
--webserver_port=0
--tserver_master_addrs=127.25.94.62:37751
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.2
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786143361779964-25976-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 22:57:40.919826 8797 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:57:40.921696 8797 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:57:40.935101 8808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:57:40.935173 8806 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:57:40.935142 8811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:57:40.936269 8797 server_base.cc:1034] running on GCE node
I20260807 22:57:40.936975 8797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:57:40.938014 8797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:57:40.939227 8797 hybrid_clock.cc:648] HybridClock initialized: now 1786143460939186 us; error 32 us; skew 500 ppm
I20260807 22:57:40.939595 8797 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 22:57:40.942061 8797 webserver.cc:460] Webserver started at http://127.25.94.2:36011/ using document root <none> and password file <none>
I20260807 22:57:40.942798 8797 fs_manager.cc:359] Metadata directory not provided
I20260807 22:57:40.942917 8797 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:57:40.943266 8797 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:57:40.946170 8797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786143361779964-25976-0/minicluster-data/ts-1/data/instance:
uuid: "e7596c6497cb423ca3422b8bc62396b5"
format_stamp: "Formatted at 2026-08-07 22:57:40 on dist-test-slave-z8x9"
server_key: "41449b717ad02ede691b38b09e895c3d"
server_key_iv: "e7649b8cba7a3e7653521f1be3030712"
server_key_version: "encryptionkey@0"
I20260807 22:57:40.946971 8797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786143361779964-25976-0/minicluster-data/ts-1/wal/instance:
uuid: "e7596c6497cb423ca3422b8bc62396b5"
format_stamp: "Formatted at 2026-08-07 22:57:40 on dist-test-slave-z8x9"
server_key: "41449b717ad02ede691b38b09e895c3d"
server_key_iv: "e7649b8cba7a3e7653521f1be3030712"
server_key_version: "encryptionkey@0"
I20260807 22:57:40.952178 8797 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260807 22:57:40.955859 8821 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:57:40.957597 8797 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 22:57:40.957808 8797 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786143361779964-25976-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786143361779964-25976-0/minicluster-data/ts-1/wal
uuid: "e7596c6497cb423ca3422b8bc62396b5"
format_stamp: "Formatted at 2026-08-07 22:57:40 on dist-test-slave-z8x9"
server_key: "41449b717ad02ede691b38b09e895c3d"
server_key_iv: "e7649b8cba7a3e7653521f1be3030712"
server_key_version: "encryptionkey@0"
I20260807 22:57:40.958047 8797 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786143361779964-25976-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786143361779964-25976-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786143361779964-25976-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:57:40.977967 8797 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:57:40.978974 8797 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:57:40.979274 8797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:57:40.980300 8797 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:57:40.982190 8797 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:57:40.982311 8797 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:57:40.982429 8797 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:57:40.982547 8797 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:57:41.017953 8797 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.2:44021
I20260807 22:57:41.018013 9002 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.2:44021 every 8 connection(s)
I20260807 22:57:41.019330 8797 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786143361779964-25976-0/minicluster-data/ts-1/data/info.pb
I20260807 22:57:41.027166 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 8797
I20260807 22:57:41.027395 25976 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786143361779964-25976-0/minicluster-data/ts-1/wal/instance
I20260807 22:57:41.027849 25976 external_mini_cluster.cc:1428] Setting key 6b6eb15b50fa04f44331129ab4a37617
I20260807 22:57:41.030997 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786143361779964-25976-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786143361779964-25976-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786143361779964-25976-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786143361779964-25976-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.3:0
--local_ip_for_outbound_sockets=127.25.94.3
--webserver_interface=127.25.94.3
--webserver_port=0
--tserver_master_addrs=127.25.94.62:37751
--builtin_ntp_servers=127.25.94.20:45677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260807 22:57:41.032759 8794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:41.022036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.2:53881 (local address 127.25.94.62:37751)
0807 22:57:41.022257 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:41.022262 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:41.023770 (+ 1508us) server_negotiation.cc:407] Connection header received
0807 22:57:41.025576 (+ 1806us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:41.025584 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:41.025673 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:41.025783 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:41.026839 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:41.027643 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:41.029714 (+ 2071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:41.030309 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:41.030453 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:41.031785 (+ 1332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:57:41.031794 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:57:41.031803 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 22:57:41.031828 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:57:41.031874 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 22:57:41.031904 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:57:41.032022 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0807 22:57:41.032227 (+ 205us) server_negotiation.cc:299] Negotiation successful
0807 22:57:41.032340 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260807 22:57:41.033608 9010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:41.022511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:37751 (local address 127.25.94.2:53881)
0807 22:57:41.023600 (+ 1089us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:41.023642 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0807 22:57:41.025300 (+ 1658us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:57:41.026005 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:57:41.026013 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:57:41.026094 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:57:41.026648 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:41.026664 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:41.027796 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:41.027801 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:41.029487 (+ 1686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:41.029502 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:41.030530 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:41.030535 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:41.030724 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:41.031451 (+ 727us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 22:57:41.031453 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:57:41.031493 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 22:57:41.031572 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:57:41.032023 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:57:41.032031 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:57:41.032037 (+ 6us) client_negotiation.cc:769] Sending connection context
0807 22:57:41.032174 (+ 137us) client_negotiation.cc:240] Negotiation successful
0807 22:57:41.032474 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614,"thread_start_us":351,"threads_started":1}
I20260807 22:57:41.035028 9003 heartbeater.cc:347] Connected to a master server at 127.25.94.62:37751
I20260807 22:57:41.035358 9003 heartbeater.cc:464] Registering TS with master...
I20260807 22:57:41.036166 9003 heartbeater.cc:511] Master 127.25.94.62:37751 requested a full tablet report, sending...
I20260807 22:57:41.037595 8495 ts_manager.cc:194] Registered new tserver with Master: e7596c6497cb423ca3422b8bc62396b5 (127.25.94.2:44021)
I20260807 22:57:41.038408 8495 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.94.2:53881
W20260807 22:57:41.387399 9011 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:57:41.387692 9011 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:57:41.387816 9011 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:57:41.395395 9011 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:57:41.395524 9011 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:57:41.395630 9011 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.3
I20260807 22:57:41.405231 9011 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:45677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786143361779964-25976-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786143361779964-25976-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786143361779964-25976-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.94.3
--webserver_port=0
--tserver_master_addrs=127.25.94.62:37751
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.3
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786143361779964-25976-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 22:57:41.406522 9011 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:57:41.408373 9011 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:57:41.422325 9017 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:57:41.422258 9020 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:57:41.422230 9018 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:57:41.423542 9011 server_base.cc:1034] running on GCE node
I20260807 22:57:41.424263 9011 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:57:41.425381 9011 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:57:41.426625 9011 hybrid_clock.cc:648] HybridClock initialized: now 1786143461426565 us; error 66 us; skew 500 ppm
I20260807 22:57:41.426973 9011 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 22:57:41.429453 9011 webserver.cc:460] Webserver started at http://127.25.94.3:44471/ using document root <none> and password file <none>
I20260807 22:57:41.430207 9011 fs_manager.cc:359] Metadata directory not provided
I20260807 22:57:41.430354 9011 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:57:41.430698 9011 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:57:41.433668 9011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786143361779964-25976-0/minicluster-data/ts-2/data/instance:
uuid: "79f3743792bc46eea64a5666908828de"
format_stamp: "Formatted at 2026-08-07 22:57:41 on dist-test-slave-z8x9"
server_key: "e9a1ca7925cd6d1b8bbdc8dc4bd8c21b"
server_key_iv: "d6342913df40b3195b8977c00b0e171b"
server_key_version: "encryptionkey@0"
I20260807 22:57:41.434489 9011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786143361779964-25976-0/minicluster-data/ts-2/wal/instance:
uuid: "79f3743792bc46eea64a5666908828de"
format_stamp: "Formatted at 2026-08-07 22:57:41 on dist-test-slave-z8x9"
server_key: "e9a1ca7925cd6d1b8bbdc8dc4bd8c21b"
server_key_iv: "d6342913df40b3195b8977c00b0e171b"
server_key_version: "encryptionkey@0"
I20260807 22:57:41.439954 9011 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260807 22:57:41.443377 9031 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:57:41.444638 9011 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260807 22:57:41.444823 9011 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786143361779964-25976-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786143361779964-25976-0/minicluster-data/ts-2/wal
uuid: "79f3743792bc46eea64a5666908828de"
format_stamp: "Formatted at 2026-08-07 22:57:41 on dist-test-slave-z8x9"
server_key: "e9a1ca7925cd6d1b8bbdc8dc4bd8c21b"
server_key_iv: "d6342913df40b3195b8977c00b0e171b"
server_key_version: "encryptionkey@0"
I20260807 22:57:41.445055 9011 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786143361779964-25976-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786143361779964-25976-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786143361779964-25976-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:57:41.459439 9011 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:57:41.460282 9011 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:57:41.460588 9011 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:57:41.461757 9011 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:57:41.463339 9011 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:57:41.463459 9011 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:57:41.463600 9011 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:57:41.463696 9011 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:57:41.498694 9011 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.3:38831
I20260807 22:57:41.498739 9212 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.3:38831 every 8 connection(s)
I20260807 22:57:41.500092 9011 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786143361779964-25976-0/minicluster-data/ts-2/data/info.pb
I20260807 22:57:41.510538 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 9011
I20260807 22:57:41.510730 25976 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786143361779964-25976-0/minicluster-data/ts-2/wal/instance
I20260807 22:57:41.511267 25976 external_mini_cluster.cc:1428] Setting key c38be0530fe74731a197e2f661f2e831
I20260807 22:57:41.512704 8794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:41.502742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.3:41061 (local address 127.25.94.62:37751)
0807 22:57:41.502921 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:41.502926 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:41.504358 (+ 1432us) server_negotiation.cc:407] Connection header received
0807 22:57:41.506143 (+ 1785us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:41.506149 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:41.506237 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:41.506360 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:41.507389 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:41.508134 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:41.509721 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:41.510188 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:41.510347 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:41.511860 (+ 1513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:57:41.511865 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:57:41.511869 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 22:57:41.511886 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:57:41.511918 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 22:57:41.511930 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:57:41.512027 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0807 22:57:41.512256 (+ 229us) server_negotiation.cc:299] Negotiation successful
0807 22:57:41.512369 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260807 22:57:41.513733 9220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:41.503155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:37751 (local address 127.25.94.3:41061)
0807 22:57:41.504187 (+ 1032us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:41.504229 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0807 22:57:41.505879 (+ 1650us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:57:41.506575 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:57:41.506583 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:57:41.506668 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:57:41.507216 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:41.507232 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:41.508281 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:41.508290 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:41.509578 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:41.509590 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:41.510430 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:41.510436 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:41.510655 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:41.511538 (+ 883us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 22:57:41.511541 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:57:41.511578 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 22:57:41.511653 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:57:41.512028 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:57:41.512041 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:57:41.512051 (+ 10us) client_negotiation.cc:769] Sending connection context
0807 22:57:41.512238 (+ 187us) client_negotiation.cc:240] Negotiation successful
0807 22:57:41.512621 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":348,"threads_started":1}
I20260807 22:57:41.515056 9216 heartbeater.cc:347] Connected to a master server at 127.25.94.62:37751
I20260807 22:57:41.515359 9216 heartbeater.cc:464] Registering TS with master...
I20260807 22:57:41.516120 9216 heartbeater.cc:511] Master 127.25.94.62:37751 requested a full tablet report, sending...
I20260807 22:57:41.517498 8495 ts_manager.cc:194] Registered new tserver with Master: 79f3743792bc46eea64a5666908828de (127.25.94.3:38831)
I20260807 22:57:41.518268 8495 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.94.3:41061
I20260807 22:57:41.526558 25976 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260807 22:57:41.533129 8794 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:57:41.531306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44838 (local address 127.25.94.62:37751)
0807 22:57:41.531613 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:41.531619 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:41.532067 (+ 448us) server_negotiation.cc:407] Connection header received
0807 22:57:41.532186 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:41.532192 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:41.532293 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:41.532384 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:41.532858 (+ 474us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44838: BlockingRecv error: recv got EOF from 127.0.0.1:44838 (error 108)
Metrics: {"server-negotiator.queue_time_us":115}
I20260807 22:57:41.533891 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 8593
I20260807 22:57:41.542039 9235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:41.534900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37848 (local address 127.25.94.1:39029)
0807 22:57:41.535401 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:41.535408 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:41.535435 (+ 27us) server_negotiation.cc:407] Connection header received
0807 22:57:41.535502 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:41.535508 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:41.535716 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:41.535860 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:41.536241 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:41.537384 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:41.538614 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:41.539163 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:41.539267 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:41.540441 (+ 1174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:57:41.540466 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:57:41.540485 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 22:57:41.541082 (+ 597us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:57:41.541129 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 22:57:41.541151 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:57:41.541350 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0807 22:57:41.541504 (+ 154us) server_negotiation.cc:299] Negotiation successful
0807 22:57:41.541668 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":251,"threads_started":1}
I20260807 22:57:41.543314 8785 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 22:57:41.625844 8792 heartbeater.cc:503] Master 127.25.94.62:37751 was elected leader, sending a full tablet report...
I20260807 22:57:41.626973 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 8593
W20260807 22:57:41.648689 8469 connection.cc:535] server connection from 127.25.94.1:35013 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260807 22:57:41.649032 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 8797
I20260807 22:57:41.656683 9237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:41.650099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48148 (local address 127.25.94.2:44021)
0807 22:57:41.650636 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:41.650644 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:41.650669 (+ 25us) server_negotiation.cc:407] Connection header received
0807 22:57:41.650729 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:41.650736 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:41.650949 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:41.651073 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:41.651459 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:41.652448 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:41.653738 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:41.654275 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:41.654392 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:41.655361 (+ 969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:57:41.655370 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:57:41.655383 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 22:57:41.655961 (+ 578us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:57:41.656005 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 22:57:41.656025 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:57:41.656116 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0807 22:57:41.656260 (+ 144us) server_negotiation.cc:299] Negotiation successful
0807 22:57:41.656385 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":274,"threads_started":1}
I20260807 22:57:41.658078 8997 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 22:57:41.740520 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 8797
I20260807 22:57:41.764637 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 9011
I20260807 22:57:41.773128 9239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:41.765632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32958 (local address 127.25.94.3:38831)
0807 22:57:41.766247 (+ 615us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:41.766255 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:41.766286 (+ 31us) server_negotiation.cc:407] Connection header received
0807 22:57:41.766372 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:41.766379 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:41.766594 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:41.766775 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:41.767245 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:41.768285 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:41.769737 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:41.770307 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:41.770447 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:41.771700 (+ 1253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:57:41.771709 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:57:41.771721 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 22:57:41.772286 (+ 565us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:57:41.772346 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 22:57:41.772365 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:57:41.772491 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0807 22:57:41.772641 (+ 150us) server_negotiation.cc:299] Negotiation successful
0807 22:57:41.772782 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":457,"thread_start_us":308,"threads_started":1}
I20260807 22:57:41.774381 9206 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 22:57:41.858562 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 9011
I20260807 22:57:41.885008 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 8444
I20260807 22:57:41.886597 8550 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 22:57:41.990049 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 8444
2026-08-07T22:57:42Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2573 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-07T22:57:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T22:57:42Z Disabled control of system clock
I20260807 22:57:42.081425 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786143361779964-25976-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786143361779964-25976-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786143361779964-25976-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.62:32831
--webserver_interface=127.25.94.62
--webserver_port=0
--builtin_ntp_servers=127.25.94.20:43395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.94.62:32831
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260807 22:57:42.436499 9254 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:57:42.436801 9254 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:57:42.436899 9254 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:57:42.444267 9254 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 22:57:42.444387 9254 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:57:42.444425 9254 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:57:42.444475 9254 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 22:57:42.444522 9254 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 22:57:42.453891 9254 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:43395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786143361779964-25976-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786143361779964-25976-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.94.62:32831
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.62:32831
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.94.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786143361779964-25976-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 22:57:42.455180 9254 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:57:42.456905 9254 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:57:42.468931 9262 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:57:42.469010 9266 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:57:42.468971 9263 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:57:42.469802 9254 server_base.cc:1034] running on GCE node
I20260807 22:57:42.470659 9254 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:57:42.472065 9254 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:57:42.473279 9254 hybrid_clock.cc:648] HybridClock initialized: now 1786143462473206 us; error 51 us; skew 500 ppm
I20260807 22:57:42.473676 9254 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 22:57:42.481151 9254 webserver.cc:460] Webserver started at http://127.25.94.62:42653/ using document root <none> and password file <none>
I20260807 22:57:42.482240 9254 fs_manager.cc:359] Metadata directory not provided
I20260807 22:57:42.482445 9254 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:57:42.482832 9254 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:57:42.485970 9254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786143361779964-25976-0/minicluster-data/master-0/data/instance:
uuid: "0257275bdfc0461d87afc77db1efb799"
format_stamp: "Formatted at 2026-08-07 22:57:42 on dist-test-slave-z8x9"
server_key: "ad0c5fca7f6be3ce4b1b25e4f7dec148"
server_key_iv: "4a92b71adc3ec16a314e1353fb2818e9"
server_key_version: "encryptionkey@0"
I20260807 22:57:42.486795 9254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786143361779964-25976-0/minicluster-data/master-0/wal/instance:
uuid: "0257275bdfc0461d87afc77db1efb799"
format_stamp: "Formatted at 2026-08-07 22:57:42 on dist-test-slave-z8x9"
server_key: "ad0c5fca7f6be3ce4b1b25e4f7dec148"
server_key_iv: "4a92b71adc3ec16a314e1353fb2818e9"
server_key_version: "encryptionkey@0"
I20260807 22:57:42.492136 9254 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260807 22:57:42.495900 9276 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:57:42.497391 9254 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 22:57:42.497579 9254 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786143361779964-25976-0/minicluster-data/master-0/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786143361779964-25976-0/minicluster-data/master-0/wal
uuid: "0257275bdfc0461d87afc77db1efb799"
format_stamp: "Formatted at 2026-08-07 22:57:42 on dist-test-slave-z8x9"
server_key: "ad0c5fca7f6be3ce4b1b25e4f7dec148"
server_key_iv: "4a92b71adc3ec16a314e1353fb2818e9"
server_key_version: "encryptionkey@0"
I20260807 22:57:42.497823 9254 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786143361779964-25976-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786143361779964-25976-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786143361779964-25976-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:57:42.503599 9254 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:57:42.504441 9254 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:57:42.504769 9254 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:57:42.524207 9254 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.62:32831
I20260807 22:57:42.524246 9360 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.62:32831 every 8 connection(s)
I20260807 22:57:42.525619 9254 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
I20260807 22:57:42.529330 9361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:57:42.530107 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 9254
I20260807 22:57:42.530361 25976 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786143361779964-25976-0/minicluster-data/master-0/wal/instance
I20260807 22:57:42.531188 25976 external_mini_cluster.cc:1428] Setting key 872675e05541c9e461310fceddf4eb62
I20260807 22:57:42.539892 9369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:42.533392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60896 (local address 127.25.94.62:32831)
0807 22:57:42.534435 (+ 1043us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:42.534448 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:42.534490 (+ 42us) server_negotiation.cc:407] Connection header received
0807 22:57:42.536052 (+ 1562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:42.536073 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:42.536482 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:42.536817 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:42.537989 (+ 1172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:57:42.537995 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:57:42.538007 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 22:57:42.538022 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:57:42.538083 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 22:57:42.538099 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:57:42.538266 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0807 22:57:42.538399 (+ 133us) server_negotiation.cc:299] Negotiation successful
0807 22:57:42.538686 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":629,"thread_start_us":405,"threads_started":1}
I20260807 22:57:42.544036 9361 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0257275bdfc0461d87afc77db1efb799: Bootstrap starting.
I20260807 22:57:42.547480 9361 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0257275bdfc0461d87afc77db1efb799: Neither blocks nor log segments found. Creating new log.
I20260807 22:57:42.548643 9361 log.cc:824] T 00000000000000000000000000000000 P 0257275bdfc0461d87afc77db1efb799: Log is configured to *not* fsync() on all Append() calls
I20260807 22:57:42.551565 9361 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0257275bdfc0461d87afc77db1efb799: No bootstrap required, opened a new log
I20260807 22:57:42.559024 9361 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0257275bdfc0461d87afc77db1efb799 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0257275bdfc0461d87afc77db1efb799" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 32831 } }
I20260807 22:57:42.559510 9361 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0257275bdfc0461d87afc77db1efb799 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:57:42.559610 9361 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0257275bdfc0461d87afc77db1efb799 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0257275bdfc0461d87afc77db1efb799, State: Initialized, Role: FOLLOWER
I20260807 22:57:42.560199 9361 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0257275bdfc0461d87afc77db1efb799 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0257275bdfc0461d87afc77db1efb799" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 32831 } }
I20260807 22:57:42.560426 9361 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0257275bdfc0461d87afc77db1efb799 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 22:57:42.560530 9361 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0257275bdfc0461d87afc77db1efb799 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 22:57:42.560642 9361 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0257275bdfc0461d87afc77db1efb799 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:57:42.562853 9361 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0257275bdfc0461d87afc77db1efb799 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0257275bdfc0461d87afc77db1efb799" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 32831 } }
I20260807 22:57:42.563306 9361 leader_election.cc:302] T 00000000000000000000000000000000 P 0257275bdfc0461d87afc77db1efb799 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0257275bdfc0461d87afc77db1efb799; no voters:
I20260807 22:57:42.563910 9361 leader_election.cc:288] T 00000000000000000000000000000000 P 0257275bdfc0461d87afc77db1efb799 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 22:57:42.564045 9374 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0257275bdfc0461d87afc77db1efb799 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:57:42.564950 9374 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0257275bdfc0461d87afc77db1efb799 [term 1 LEADER]: Becoming Leader. State: Replica: 0257275bdfc0461d87afc77db1efb799, State: Running, Role: LEADER
I20260807 22:57:42.565512 9374 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0257275bdfc0461d87afc77db1efb799 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0257275bdfc0461d87afc77db1efb799" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 32831 } }
I20260807 22:57:42.566179 9361 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0257275bdfc0461d87afc77db1efb799 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 22:57:42.568228 9379 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0257275bdfc0461d87afc77db1efb799 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0257275bdfc0461d87afc77db1efb799. Latest consensus state: current_term: 1 leader_uuid: "0257275bdfc0461d87afc77db1efb799" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0257275bdfc0461d87afc77db1efb799" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 32831 } } }
I20260807 22:57:42.568177 9375 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0257275bdfc0461d87afc77db1efb799 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0257275bdfc0461d87afc77db1efb799" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0257275bdfc0461d87afc77db1efb799" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 32831 } } }
I20260807 22:57:42.568616 9379 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0257275bdfc0461d87afc77db1efb799 [sys.catalog]: This master's current role is: LEADER
I20260807 22:57:42.568612 9375 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0257275bdfc0461d87afc77db1efb799 [sys.catalog]: This master's current role is: LEADER
I20260807 22:57:42.570910 9390 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 22:57:42.576018 9390 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 22:57:42.585489 9390 catalog_manager.cc:1347] Generated new cluster ID: 45d9b4133e604af6b964f69bfaf753cf
I20260807 22:57:42.585621 9390 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 22:57:42.603677 9390 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 22:57:42.604218 9390 catalog_manager.cc:1504] Loading token signing keys...
I20260807 22:57:42.611734 9390 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0257275bdfc0461d87afc77db1efb799: Generated new TSK 0
I20260807 22:57:42.612670 9390 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 22:57:42.619765 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786143361779964-25976-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786143361779964-25976-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786143361779964-25976-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786143361779964-25976-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.1:0
--local_ip_for_outbound_sockets=127.25.94.1
--webserver_interface=127.25.94.1
--webserver_port=0
--tserver_master_addrs=127.25.94.62:32831
--builtin_ntp_servers=127.25.94.20:43395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260807 22:57:42.992399 9412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:57:42.992741 9412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:57:42.992920 9412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:57:43.000824 9412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:57:43.000977 9412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:57:43.001160 9412 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.1
I20260807 22:57:43.010684 9412 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:43395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786143361779964-25976-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786143361779964-25976-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786143361779964-25976-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.94.1
--webserver_port=0
--tserver_master_addrs=127.25.94.62:32831
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.1
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786143361779964-25976-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 22:57:43.012012 9412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:57:43.013921 9412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:57:43.027493 9421 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:57:43.027449 9425 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:57:43.027547 9420 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:57:43.030898 9412 server_base.cc:1034] running on GCE node
I20260807 22:57:43.031584 9412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:57:43.032595 9412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:57:43.033818 9412 hybrid_clock.cc:648] HybridClock initialized: now 1786143463033761 us; error 36 us; skew 500 ppm
I20260807 22:57:43.034157 9412 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 22:57:43.036537 9412 webserver.cc:460] Webserver started at http://127.25.94.1:37243/ using document root <none> and password file <none>
I20260807 22:57:43.037343 9412 fs_manager.cc:359] Metadata directory not provided
I20260807 22:57:43.037462 9412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:57:43.037806 9412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:57:43.043356 9412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786143361779964-25976-0/minicluster-data/ts-0/data/instance:
uuid: "307994c1cfa747389af18f1d5f823631"
format_stamp: "Formatted at 2026-08-07 22:57:43 on dist-test-slave-z8x9"
server_key: "4c4f6b5507d2b493f6ea1788f52f3769"
server_key_iv: "908978975c1d0e21b6baa575078cb972"
server_key_version: "encryptionkey@0"
I20260807 22:57:43.044178 9412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786143361779964-25976-0/minicluster-data/ts-0/wal/instance:
uuid: "307994c1cfa747389af18f1d5f823631"
format_stamp: "Formatted at 2026-08-07 22:57:43 on dist-test-slave-z8x9"
server_key: "4c4f6b5507d2b493f6ea1788f52f3769"
server_key_iv: "908978975c1d0e21b6baa575078cb972"
server_key_version: "encryptionkey@0"
I20260807 22:57:43.049448 9412 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260807 22:57:43.052932 9434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:57:43.054466 9412 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 22:57:43.054665 9412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786143361779964-25976-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786143361779964-25976-0/minicluster-data/ts-0/wal
uuid: "307994c1cfa747389af18f1d5f823631"
format_stamp: "Formatted at 2026-08-07 22:57:43 on dist-test-slave-z8x9"
server_key: "4c4f6b5507d2b493f6ea1788f52f3769"
server_key_iv: "908978975c1d0e21b6baa575078cb972"
server_key_version: "encryptionkey@0"
I20260807 22:57:43.054906 9412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786143361779964-25976-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786143361779964-25976-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786143361779964-25976-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:57:43.061039 9412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:57:43.061933 9412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:57:43.062168 9412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:57:43.063050 9412 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:57:43.064545 9412 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:57:43.064638 9412 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:57:43.064723 9412 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:57:43.064781 9412 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:57:43.100550 9412 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.1:41633
I20260807 22:57:43.100641 9640 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.1:41633 every 8 connection(s)
I20260807 22:57:43.102159 9412 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786143361779964-25976-0/minicluster-data/ts-0/data/info.pb
I20260807 22:57:43.111532 9646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:43.105295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.1:43779 (local address 127.25.94.62:32831)
0807 22:57:43.105829 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:43.105833 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:43.107196 (+ 1363us) server_negotiation.cc:407] Connection header received
0807 22:57:43.108979 (+ 1783us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:43.108986 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:43.109072 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:43.109221 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:43.110673 (+ 1452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:57:43.110679 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:57:43.110684 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 22:57:43.110704 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:57:43.110756 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 22:57:43.110767 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:57:43.110915 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0807 22:57:43.111132 (+ 217us) server_negotiation.cc:299] Negotiation successful
0807 22:57:43.111224 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":287,"threads_started":1}
I20260807 22:57:43.112471 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 9412
I20260807 22:57:43.112613 9647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:43.105728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:32831 (local address 127.25.94.1:43779)
0807 22:57:43.107033 (+ 1305us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:43.107082 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0807 22:57:43.108687 (+ 1605us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:57:43.109493 (+ 806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:57:43.109502 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:57:43.109601 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:57:43.110362 (+ 761us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 22:57:43.110365 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:57:43.110404 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 22:57:43.110495 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:57:43.110873 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:57:43.110884 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:57:43.110893 (+ 9us) client_negotiation.cc:769] Sending connection context
0807 22:57:43.111223 (+ 330us) client_negotiation.cc:240] Negotiation successful
0807 22:57:43.111494 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":597,"thread_start_us":367,"threads_started":1}
I20260807 22:57:43.112720 25976 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786143361779964-25976-0/minicluster-data/ts-0/wal/instance
I20260807 22:57:43.113580 25976 external_mini_cluster.cc:1428] Setting key 6665417f2df89eb9dcc03da2df051d43
I20260807 22:57:43.114391 9643 heartbeater.cc:347] Connected to a master server at 127.25.94.62:32831
I20260807 22:57:43.114771 9643 heartbeater.cc:464] Registering TS with master...
I20260807 22:57:43.115528 9643 heartbeater.cc:511] Master 127.25.94.62:32831 requested a full tablet report, sending...
I20260807 22:57:43.117247 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786143361779964-25976-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786143361779964-25976-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786143361779964-25976-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786143361779964-25976-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.2:0
--local_ip_for_outbound_sockets=127.25.94.2
--webserver_interface=127.25.94.2
--webserver_port=0
--tserver_master_addrs=127.25.94.62:32831
--builtin_ntp_servers=127.25.94.20:43395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260807 22:57:43.118085 9307 ts_manager.cc:194] Registered new tserver with Master: 307994c1cfa747389af18f1d5f823631 (127.25.94.1:41633)
W20260807 22:57:43.471985 9648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:57:43.472334 9648 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:57:43.472504 9648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:57:43.479918 9648 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:57:43.480073 9648 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:57:43.480262 9648 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.2
I20260807 22:57:43.489600 9648 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:43395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786143361779964-25976-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786143361779964-25976-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786143361779964-25976-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.94.2
--webserver_port=0
--tserver_master_addrs=127.25.94.62:32831
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.2
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786143361779964-25976-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 22:57:43.490963 9648 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:57:43.492834 9648 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:57:43.507074 9657 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:57:43.507035 9661 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:57:43.507287 9659 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:57:43.507781 9648 server_base.cc:1034] running on GCE node
I20260807 22:57:43.508519 9648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:57:43.509716 9648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:57:43.510951 9648 hybrid_clock.cc:648] HybridClock initialized: now 1786143463510900 us; error 61 us; skew 500 ppm
I20260807 22:57:43.511274 9648 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 22:57:43.513707 9648 webserver.cc:460] Webserver started at http://127.25.94.2:44771/ using document root <none> and password file <none>
I20260807 22:57:43.514436 9648 fs_manager.cc:359] Metadata directory not provided
I20260807 22:57:43.514569 9648 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:57:43.514900 9648 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:57:43.517794 9648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786143361779964-25976-0/minicluster-data/ts-1/data/instance:
uuid: "aaa69d3f2e6d4246bf95fe97b79165f6"
format_stamp: "Formatted at 2026-08-07 22:57:43 on dist-test-slave-z8x9"
server_key: "1056a44483c020bfb14a075888fb515f"
server_key_iv: "20803ff968484699203d6657d7537693"
server_key_version: "encryptionkey@0"
I20260807 22:57:43.518584 9648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786143361779964-25976-0/minicluster-data/ts-1/wal/instance:
uuid: "aaa69d3f2e6d4246bf95fe97b79165f6"
format_stamp: "Formatted at 2026-08-07 22:57:43 on dist-test-slave-z8x9"
server_key: "1056a44483c020bfb14a075888fb515f"
server_key_iv: "20803ff968484699203d6657d7537693"
server_key_version: "encryptionkey@0"
I20260807 22:57:43.523677 9648 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260807 22:57:43.527244 9668 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:57:43.528573 9648 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 22:57:43.528746 9648 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786143361779964-25976-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786143361779964-25976-0/minicluster-data/ts-1/wal
uuid: "aaa69d3f2e6d4246bf95fe97b79165f6"
format_stamp: "Formatted at 2026-08-07 22:57:43 on dist-test-slave-z8x9"
server_key: "1056a44483c020bfb14a075888fb515f"
server_key_iv: "20803ff968484699203d6657d7537693"
server_key_version: "encryptionkey@0"
I20260807 22:57:43.528965 9648 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786143361779964-25976-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786143361779964-25976-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786143361779964-25976-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:57:43.534683 9648 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:57:43.535458 9648 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:57:43.535737 9648 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:57:43.536672 9648 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:57:43.538153 9648 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:57:43.538272 9648 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:57:43.538409 9648 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:57:43.538497 9648 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:57:43.571985 9648 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.2:33641
I20260807 22:57:43.572046 9878 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.2:33641 every 8 connection(s)
I20260807 22:57:43.573421 9648 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786143361779964-25976-0/minicluster-data/ts-1/data/info.pb
I20260807 22:57:43.576614 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 9648
I20260807 22:57:43.576808 25976 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786143361779964-25976-0/minicluster-data/ts-1/wal/instance
I20260807 22:57:43.577390 25976 external_mini_cluster.cc:1428] Setting key 3a7c8e6ea9ea0a959b602d72a2d17b75
I20260807 22:57:43.580227 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786143361779964-25976-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786143361779964-25976-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786143361779964-25976-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786143361779964-25976-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.3:0
--local_ip_for_outbound_sockets=127.25.94.3
--webserver_interface=127.25.94.3
--webserver_port=0
--tserver_master_addrs=127.25.94.62:32831
--builtin_ntp_servers=127.25.94.20:43395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260807 22:57:43.582249 9646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:43.576041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.2:51317 (local address 127.25.94.62:32831)
0807 22:57:43.576224 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:43.576229 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:43.577836 (+ 1607us) server_negotiation.cc:407] Connection header received
0807 22:57:43.579754 (+ 1918us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:43.579760 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:43.579827 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:43.579943 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:43.581412 (+ 1469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:57:43.581422 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:57:43.581430 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 22:57:43.581454 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:57:43.581498 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 22:57:43.581517 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:57:43.581645 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0807 22:57:43.581837 (+ 192us) server_negotiation.cc:299] Negotiation successful
0807 22:57:43.581904 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260807 22:57:43.583010 9890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:43.576485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:32831 (local address 127.25.94.2:51317)
0807 22:57:43.577650 (+ 1165us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:43.577707 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0807 22:57:43.579501 (+ 1794us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:57:43.580124 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:57:43.580131 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:57:43.580218 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:57:43.581026 (+ 808us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 22:57:43.581029 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:57:43.581065 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 22:57:43.581152 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:57:43.581623 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:57:43.581631 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:57:43.581637 (+ 6us) client_negotiation.cc:769] Sending connection context
0807 22:57:43.581869 (+ 232us) client_negotiation.cc:240] Negotiation successful
0807 22:57:43.582056 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":399,"threads_started":1}
I20260807 22:57:43.584484 9883 heartbeater.cc:347] Connected to a master server at 127.25.94.62:32831
I20260807 22:57:43.584800 9883 heartbeater.cc:464] Registering TS with master...
I20260807 22:57:43.585474 9883 heartbeater.cc:511] Master 127.25.94.62:32831 requested a full tablet report, sending...
I20260807 22:57:43.587040 9307 ts_manager.cc:194] Registered new tserver with Master: aaa69d3f2e6d4246bf95fe97b79165f6 (127.25.94.2:33641)
W20260807 22:57:43.922988 9892 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:57:43.923283 9892 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:57:43.923377 9892 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:57:43.930730 9892 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:57:43.930831 9892 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:57:43.930982 9892 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.3
I20260807 22:57:43.940392 9892 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:43395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786143361779964-25976-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786143361779964-25976-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786143361779964-25976-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.94.3
--webserver_port=0
--tserver_master_addrs=127.25.94.62:32831
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.3
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786143361779964-25976-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 22:57:43.941807 9892 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:57:43.943645 9892 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:57:43.957129 9900 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:57:43.957196 9901 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:57:43.957093 9904 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:57:43.958110 9892 server_base.cc:1034] running on GCE node
I20260807 22:57:43.958781 9892 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:57:43.959761 9892 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:57:43.960971 9892 hybrid_clock.cc:648] HybridClock initialized: now 1786143463960898 us; error 64 us; skew 500 ppm
I20260807 22:57:43.961347 9892 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 22:57:43.963697 9892 webserver.cc:460] Webserver started at http://127.25.94.3:42519/ using document root <none> and password file <none>
I20260807 22:57:43.964398 9892 fs_manager.cc:359] Metadata directory not provided
I20260807 22:57:43.964490 9892 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:57:43.964758 9892 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:57:43.967921 9892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786143361779964-25976-0/minicluster-data/ts-2/data/instance:
uuid: "c8ba16f98c274a43a21a5b62ff41b852"
format_stamp: "Formatted at 2026-08-07 22:57:43 on dist-test-slave-z8x9"
server_key: "2c788bf1c7693e5d3a85e4ffc5dc9b0b"
server_key_iv: "65a577cdfcc1058fb518c5272b92f10d"
server_key_version: "encryptionkey@0"
I20260807 22:57:43.968690 9892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786143361779964-25976-0/minicluster-data/ts-2/wal/instance:
uuid: "c8ba16f98c274a43a21a5b62ff41b852"
format_stamp: "Formatted at 2026-08-07 22:57:43 on dist-test-slave-z8x9"
server_key: "2c788bf1c7693e5d3a85e4ffc5dc9b0b"
server_key_iv: "65a577cdfcc1058fb518c5272b92f10d"
server_key_version: "encryptionkey@0"
I20260807 22:57:43.974048 9892 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260807 22:57:43.977478 9914 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:57:43.978859 9892 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260807 22:57:43.978993 9892 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786143361779964-25976-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786143361779964-25976-0/minicluster-data/ts-2/wal
uuid: "c8ba16f98c274a43a21a5b62ff41b852"
format_stamp: "Formatted at 2026-08-07 22:57:43 on dist-test-slave-z8x9"
server_key: "2c788bf1c7693e5d3a85e4ffc5dc9b0b"
server_key_iv: "65a577cdfcc1058fb518c5272b92f10d"
server_key_version: "encryptionkey@0"
I20260807 22:57:43.979175 9892 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786143361779964-25976-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786143361779964-25976-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786143361779964-25976-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:57:43.984871 9892 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:57:43.985810 9892 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:57:43.986047 9892 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:57:43.986928 9892 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:57:43.988301 9892 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:57:43.988391 9892 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:57:43.988472 9892 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:57:43.988528 9892 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:57:44.023138 9892 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.3:36021
I20260807 22:57:44.023213 10129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.3:36021 every 8 connection(s)
I20260807 22:57:44.024492 9892 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786143361779964-25976-0/minicluster-data/ts-2/data/info.pb
I20260807 22:57:44.029358 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 9892
I20260807 22:57:44.029551 25976 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786143361779964-25976-0/minicluster-data/ts-2/wal/instance
I20260807 22:57:44.029983 25976 external_mini_cluster.cc:1428] Setting key 0652a1dbed43147710afced5eff6b121
I20260807 22:57:44.033109 9646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:44.027231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.3:52837 (local address 127.25.94.62:32831)
0807 22:57:44.027409 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:44.027414 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:44.028768 (+ 1354us) server_negotiation.cc:407] Connection header received
0807 22:57:44.030671 (+ 1903us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:44.030677 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:44.030765 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:44.030892 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:44.032268 (+ 1376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:57:44.032274 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:57:44.032279 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 22:57:44.032297 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:57:44.032339 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 22:57:44.032354 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:57:44.032481 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0807 22:57:44.032713 (+ 232us) server_negotiation.cc:299] Negotiation successful
0807 22:57:44.032789 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260807 22:57:44.034173 10132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:44.027724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:32831 (local address 127.25.94.3:52837)
0807 22:57:44.028609 (+ 885us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:44.028644 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0807 22:57:44.030398 (+ 1754us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:57:44.031130 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:57:44.031138 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:57:44.031230 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:57:44.031954 (+ 724us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 22:57:44.031957 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:57:44.031996 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 22:57:44.032103 (+ 107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:57:44.032482 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:57:44.032494 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:57:44.032502 (+ 8us) client_negotiation.cc:769] Sending connection context
0807 22:57:44.032814 (+ 312us) client_negotiation.cc:240] Negotiation successful
0807 22:57:44.033089 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":286,"threads_started":1}
I20260807 22:57:44.035439 10130 heartbeater.cc:347] Connected to a master server at 127.25.94.62:32831
I20260807 22:57:44.035693 10130 heartbeater.cc:464] Registering TS with master...
I20260807 22:57:44.036248 10130 heartbeater.cc:511] Master 127.25.94.62:32831 requested a full tablet report, sending...
I20260807 22:57:44.037621 9307 ts_manager.cc:194] Registered new tserver with Master: c8ba16f98c274a43a21a5b62ff41b852 (127.25.94.3:36021)
I20260807 22:57:44.046134 25976 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260807 22:57:44.052454 9646 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:57:44.050727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60910 (local address 127.25.94.62:32831)
0807 22:57:44.050929 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:44.050938 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:44.051404 (+ 466us) server_negotiation.cc:407] Connection header received
0807 22:57:44.051605 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:44.051611 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:44.051679 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:44.051794 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:44.052167 (+ 373us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60910: BlockingRecv error: recv got EOF from 127.0.0.1:60910 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260807 22:57:44.053064 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 9412
I20260807 22:57:44.056465 10148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:44.054032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47906 (local address 127.25.94.1:41633)
0807 22:57:44.054548 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:44.054555 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:44.054580 (+ 25us) server_negotiation.cc:407] Connection header received
0807 22:57:44.054636 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:44.054641 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:44.054841 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:44.054967 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:44.055830 (+ 863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:57:44.055837 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:57:44.055848 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 22:57:44.055861 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:57:44.055902 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 22:57:44.055919 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:57:44.056023 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0807 22:57:44.056117 (+ 94us) server_negotiation.cc:299] Negotiation successful
0807 22:57:44.056203 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":240,"threads_started":1}
I20260807 22:57:44.057801 9633 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 22:57:44.141366 9643 heartbeater.cc:503] Master 127.25.94.62:32831 was elected leader, sending a full tablet report...
I20260807 22:57:44.142776 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 9412
W20260807 22:57:44.167107 9285 connection.cc:535] server connection from 127.25.94.1:43779 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260807 22:57:44.167444 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 9648
I20260807 22:57:44.171533 10150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:44.168537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54944 (local address 127.25.94.2:33641)
0807 22:57:44.169101 (+ 564us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:44.169108 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:44.169135 (+ 27us) server_negotiation.cc:407] Connection header received
0807 22:57:44.169260 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:44.169267 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:44.169509 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:44.169676 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:44.170781 (+ 1105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:57:44.170795 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:57:44.170807 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 22:57:44.170831 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:57:44.170878 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 22:57:44.170896 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:57:44.171061 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0807 22:57:44.171141 (+ 80us) server_negotiation.cc:299] Negotiation successful
0807 22:57:44.171260 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":289,"threads_started":1}
I20260807 22:57:44.173089 9867 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 22:57:44.257719 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 9648
I20260807 22:57:44.284767 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 9892
I20260807 22:57:44.288946 10153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:44.285835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52866 (local address 127.25.94.3:36021)
0807 22:57:44.286591 (+ 756us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:44.286606 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:44.286637 (+ 31us) server_negotiation.cc:407] Connection header received
0807 22:57:44.286704 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:44.286710 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:44.286926 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:44.287056 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:44.288181 (+ 1125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:57:44.288188 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:57:44.288198 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 22:57:44.288216 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:57:44.288258 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 22:57:44.288272 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:57:44.288411 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0807 22:57:44.288465 (+ 54us) server_negotiation.cc:299] Negotiation successful
0807 22:57:44.288576 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":612,"thread_start_us":423,"threads_started":1}
I20260807 22:57:44.290236 10122 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 22:57:44.373431 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 9892
I20260807 22:57:44.397584 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 9254
I20260807 22:57:44.399087 9355 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 22:57:44.518280 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 9254
2026-08-07T22:57:44Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2548 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786143361779964-25976-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 22:57:44 dist-test-slave-z8x9 krb5kdc[10160](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 22:57:44 dist-test-slave-z8x9 krb5kdc[10160](info): set up 2 sockets
Aug 07 22:57:44 dist-test-slave-z8x9 krb5kdc[10160](info): commencing operation
krb5kdc: starting...
W20260807 22:57:46.602733 25976 mini_kdc.cc:121] Time spent starting KDC: real 2.008s user 0.004s sys 0.057s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786143361779964-25976-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786143361779964-25976-0/krb5kdc/test-user.keytab.
Aug 07 22:57:46 dist-test-slave-z8x9 krb5kdc[10160](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143466, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T22:57:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T22:57:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.94.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.62.keytab.
Entry for principal kudu/127.25.94.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.62.keytab.
I20260807 22:57:47.107910 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786143361779964-25976-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786143361779964-25976-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786143361779964-25976-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.62:35335
--webserver_interface=127.25.94.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.62.keytab
--principal=kudu/127.25.94.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.20:38243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.94.62:35335
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786143361779964-25976-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786143361779964-25976-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786143361779964-25976-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:57:47.501696 10178 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:57:47.501991 10178 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:57:47.502170 10178 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:57:47.509452 10178 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 22:57:47.509562 10178 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:57:47.509675 10178 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:57:47.509783 10178 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 22:57:47.509869 10178 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 22:57:47.519660 10178 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:38243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786143361779964-25976-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786143361779964-25976-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.94.62:35335
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.62.keytab
--principal=kudu/127.25.94.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.62:35335
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.94.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786143361779964-25976-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 22:57:47.521128 10178 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:57:47.522528 10178 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:57:47.534809 10187 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:57:47.534723 10193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:57:47.534767 10190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:57:47.536278 10178 server_base.cc:1034] running on GCE node
I20260807 22:57:47.537115 10178 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:57:47.538597 10178 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:57:47.539850 10178 hybrid_clock.cc:648] HybridClock initialized: now 1786143467539811 us; error 53 us; skew 500 ppm
I20260807 22:57:47.540235 10178 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 22:57:47 dist-test-slave-z8x9 krb5kdc[10160](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143467, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:57:47.546911 10178 init.cc:377] Logged in from keytab as kudu/127.25.94.62@KRBTEST.COM (short username kudu)
I20260807 22:57:47.548542 10178 webserver.cc:460] Webserver started at http://127.25.94.62:45535/ using document root <none> and password file <none>
I20260807 22:57:47.549320 10178 fs_manager.cc:359] Metadata directory not provided
I20260807 22:57:47.549445 10178 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:57:47.549731 10178 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:57:47.552795 10178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786143361779964-25976-0/minicluster-data/master-0/data/instance:
uuid: "667233c706e44aef884f4b51f73bdd6d"
format_stamp: "Formatted at 2026-08-07 22:57:47 on dist-test-slave-z8x9"
server_key: "6f4cc95a2ce1c3d82b1652e7bcdc8ce3"
server_key_iv: "963680fc1d96bccb5c9fc7f043c7fd63"
server_key_version: "encryptionkey@0"
I20260807 22:57:47.553685 10178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786143361779964-25976-0/minicluster-data/master-0/wal/instance:
uuid: "667233c706e44aef884f4b51f73bdd6d"
format_stamp: "Formatted at 2026-08-07 22:57:47 on dist-test-slave-z8x9"
server_key: "6f4cc95a2ce1c3d82b1652e7bcdc8ce3"
server_key_iv: "963680fc1d96bccb5c9fc7f043c7fd63"
server_key_version: "encryptionkey@0"
I20260807 22:57:47.559062 10178 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260807 22:57:47.562762 10203 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:57:47.564260 10178 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260807 22:57:47.564419 10178 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786143361779964-25976-0/minicluster-data/master-0/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786143361779964-25976-0/minicluster-data/master-0/wal
uuid: "667233c706e44aef884f4b51f73bdd6d"
format_stamp: "Formatted at 2026-08-07 22:57:47 on dist-test-slave-z8x9"
server_key: "6f4cc95a2ce1c3d82b1652e7bcdc8ce3"
server_key_iv: "963680fc1d96bccb5c9fc7f043c7fd63"
server_key_version: "encryptionkey@0"
I20260807 22:57:47.564592 10178 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786143361779964-25976-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786143361779964-25976-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786143361779964-25976-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:57:47.583953 10178 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:57:47.588690 10178 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:57:47.588964 10178 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:57:47.609295 10178 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.62:35335
I20260807 22:57:47.609333 10305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.62:35335 every 8 connection(s)
I20260807 22:57:47.611274 10178 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
I20260807 22:57:47.611814 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 10178
I20260807 22:57:47.612068 25976 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786143361779964-25976-0/minicluster-data/master-0/wal/instance
I20260807 22:57:47.612932 25976 external_mini_cluster.cc:1428] Setting key 4566e37006cbe9f2013c78cd96f6a6c9
I20260807 22:57:47.616221 10307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 07 22:57:47 dist-test-slave-z8x9 krb5kdc[10160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143466, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.62@KRBTEST.COM
I20260807 22:57:47.629792 10307 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 667233c706e44aef884f4b51f73bdd6d: Bootstrap starting.
I20260807 22:57:47.633457 10307 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 667233c706e44aef884f4b51f73bdd6d: Neither blocks nor log segments found. Creating new log.
I20260807 22:57:47.634827 10307 log.cc:824] T 00000000000000000000000000000000 P 667233c706e44aef884f4b51f73bdd6d: Log is configured to *not* fsync() on all Append() calls
I20260807 22:57:47.638230 10314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:47.615235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52704 (local address 127.25.94.62:35335)
0807 22:57:47.616152 (+ 917us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:47.616169 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:47.616231 (+ 62us) server_negotiation.cc:407] Connection header received
0807 22:57:47.618635 (+ 2404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:47.618663 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:47.619158 (+ 495us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:47.619619 (+ 461us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:47.621044 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:47.623216 (+ 2172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:47.624535 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:47.625628 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:47.625823 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:47.630174 (+ 4351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:57:47.630183 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:57:47.630200 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:57:47.630238 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:57:47.633704 (+ 3466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:47.634607 (+ 903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:47.634614 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:47.634620 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:47.634792 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:47.635253 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:47.635257 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:47.635259 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:47.636051 (+ 792us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:57:47.636427 (+ 376us) server_negotiation.cc:1035] Waiting for connection context
0807 22:57:47.636801 (+ 374us) server_negotiation.cc:299] Negotiation successful
0807 22:57:47.637118 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":677,"thread_start_us":407,"threads_started":1}
I20260807 22:57:47.638427 10307 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 667233c706e44aef884f4b51f73bdd6d: No bootstrap required, opened a new log
I20260807 22:57:47.646952 10307 raft_consensus.cc:348] T 00000000000000000000000000000000 P 667233c706e44aef884f4b51f73bdd6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "667233c706e44aef884f4b51f73bdd6d" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 35335 } }
I20260807 22:57:47.647347 10307 raft_consensus.cc:374] T 00000000000000000000000000000000 P 667233c706e44aef884f4b51f73bdd6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:57:47.647478 10307 raft_consensus.cc:729] T 00000000000000000000000000000000 P 667233c706e44aef884f4b51f73bdd6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 667233c706e44aef884f4b51f73bdd6d, State: Initialized, Role: FOLLOWER
I20260807 22:57:47.648111 10307 consensus_queue.cc:260] T 00000000000000000000000000000000 P 667233c706e44aef884f4b51f73bdd6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "667233c706e44aef884f4b51f73bdd6d" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 35335 } }
I20260807 22:57:47.648332 10307 raft_consensus.cc:388] T 00000000000000000000000000000000 P 667233c706e44aef884f4b51f73bdd6d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 22:57:47.648485 10307 raft_consensus.cc:482] T 00000000000000000000000000000000 P 667233c706e44aef884f4b51f73bdd6d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 22:57:47.648669 10307 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 667233c706e44aef884f4b51f73bdd6d [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:57:47.651185 10307 raft_consensus.cc:504] T 00000000000000000000000000000000 P 667233c706e44aef884f4b51f73bdd6d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "667233c706e44aef884f4b51f73bdd6d" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 35335 } }
I20260807 22:57:47.651754 10307 leader_election.cc:302] T 00000000000000000000000000000000 P 667233c706e44aef884f4b51f73bdd6d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 667233c706e44aef884f4b51f73bdd6d; no voters:
I20260807 22:57:47.652367 10307 leader_election.cc:288] T 00000000000000000000000000000000 P 667233c706e44aef884f4b51f73bdd6d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 22:57:47.652537 10319 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 667233c706e44aef884f4b51f73bdd6d [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:57:47.653367 10319 raft_consensus.cc:686] T 00000000000000000000000000000000 P 667233c706e44aef884f4b51f73bdd6d [term 1 LEADER]: Becoming Leader. State: Replica: 667233c706e44aef884f4b51f73bdd6d, State: Running, Role: LEADER
I20260807 22:57:47.653954 10319 consensus_queue.cc:237] T 00000000000000000000000000000000 P 667233c706e44aef884f4b51f73bdd6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "667233c706e44aef884f4b51f73bdd6d" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 35335 } }
I20260807 22:57:47.654594 10307 sys_catalog.cc:564] T 00000000000000000000000000000000 P 667233c706e44aef884f4b51f73bdd6d [sys.catalog]: configured and running, proceeding with master startup.
I20260807 22:57:47.657210 10324 sys_catalog.cc:455] T 00000000000000000000000000000000 P 667233c706e44aef884f4b51f73bdd6d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 667233c706e44aef884f4b51f73bdd6d. Latest consensus state: current_term: 1 leader_uuid: "667233c706e44aef884f4b51f73bdd6d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "667233c706e44aef884f4b51f73bdd6d" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 35335 } } }
I20260807 22:57:47.657645 10324 sys_catalog.cc:458] T 00000000000000000000000000000000 P 667233c706e44aef884f4b51f73bdd6d [sys.catalog]: This master's current role is: LEADER
I20260807 22:57:47.658170 10323 sys_catalog.cc:455] T 00000000000000000000000000000000 P 667233c706e44aef884f4b51f73bdd6d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "667233c706e44aef884f4b51f73bdd6d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "667233c706e44aef884f4b51f73bdd6d" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 35335 } } }
I20260807 22:57:47.658499 10323 sys_catalog.cc:458] T 00000000000000000000000000000000 P 667233c706e44aef884f4b51f73bdd6d [sys.catalog]: This master's current role is: LEADER
I20260807 22:57:47.662125 10335 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 22:57:47.667580 10335 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 22:57:47.677009 10335 catalog_manager.cc:1347] Generated new cluster ID: a50eaca89cd34da0aa5eae826bc282bc
I20260807 22:57:47.677142 10335 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 22:57:47.696045 10335 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 22:57:47.697732 10335 catalog_manager.cc:1504] Loading token signing keys...
I20260807 22:57:47.707755 10335 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 667233c706e44aef884f4b51f73bdd6d: Generated new TSK 0
I20260807 22:57:47.708627 10335 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.94.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.1.keytab.
Entry for principal kudu/127.25.94.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.1.keytab.
I20260807 22:57:47.786062 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786143361779964-25976-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786143361779964-25976-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786143361779964-25976-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786143361779964-25976-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.1:0
--local_ip_for_outbound_sockets=127.25.94.1
--webserver_interface=127.25.94.1
--webserver_port=0
--tserver_master_addrs=127.25.94.62:35335
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.1.keytab
--principal=kudu/127.25.94.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.20:38243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786143361779964-25976-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786143361779964-25976-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786143361779964-25976-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:57:48.141804 10357 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:57:48.142140 10357 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:57:48.142297 10357 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:57:48.149961 10357 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:57:48.150095 10357 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:57:48.150276 10357 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.1
I20260807 22:57:48.162282 10357 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:38243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786143361779964-25976-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786143361779964-25976-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.1.keytab
--principal=kudu/127.25.94.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786143361779964-25976-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.94.1
--webserver_port=0
--tserver_master_addrs=127.25.94.62:35335
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.1
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786143361779964-25976-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 22:57:48.163739 10357 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:57:48.165576 10357 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:57:48.180754 10363 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:57:48.180809 10371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:57:48.180862 10365 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:57:48.181898 10357 server_base.cc:1034] running on GCE node
I20260807 22:57:48.182658 10357 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:57:48.183813 10357 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:57:48.185084 10357 hybrid_clock.cc:648] HybridClock initialized: now 1786143468185034 us; error 49 us; skew 500 ppm
I20260807 22:57:48.185478 10357 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 22:57:48 dist-test-slave-z8x9 krb5kdc[10160](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143468, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:57:48.189064 10357 init.cc:377] Logged in from keytab as kudu/127.25.94.1@KRBTEST.COM (short username kudu)
I20260807 22:57:48.190963 10357 webserver.cc:460] Webserver started at http://127.25.94.1:42549/ using document root <none> and password file <none>
I20260807 22:57:48.191843 10357 fs_manager.cc:359] Metadata directory not provided
I20260807 22:57:48.191994 10357 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:57:48.192293 10357 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:57:48.195612 10357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786143361779964-25976-0/minicluster-data/ts-0/data/instance:
uuid: "28bb0f3e219e46489ba58f36a2f92f96"
format_stamp: "Formatted at 2026-08-07 22:57:48 on dist-test-slave-z8x9"
server_key: "3585a203d6a96fb9bf783bab8db7e559"
server_key_iv: "4a1704981c21fe47b0acc5e4fd7d23f6"
server_key_version: "encryptionkey@0"
I20260807 22:57:48.197395 10357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786143361779964-25976-0/minicluster-data/ts-0/wal/instance:
uuid: "28bb0f3e219e46489ba58f36a2f92f96"
format_stamp: "Formatted at 2026-08-07 22:57:48 on dist-test-slave-z8x9"
server_key: "3585a203d6a96fb9bf783bab8db7e559"
server_key_iv: "4a1704981c21fe47b0acc5e4fd7d23f6"
server_key_version: "encryptionkey@0"
I20260807 22:57:48.203675 10357 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.003s
I20260807 22:57:48.207773 10383 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:57:48.209583 10357 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260807 22:57:48.209766 10357 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786143361779964-25976-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786143361779964-25976-0/minicluster-data/ts-0/wal
uuid: "28bb0f3e219e46489ba58f36a2f92f96"
format_stamp: "Formatted at 2026-08-07 22:57:48 on dist-test-slave-z8x9"
server_key: "3585a203d6a96fb9bf783bab8db7e559"
server_key_iv: "4a1704981c21fe47b0acc5e4fd7d23f6"
server_key_version: "encryptionkey@0"
I20260807 22:57:48.209982 10357 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786143361779964-25976-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786143361779964-25976-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786143361779964-25976-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:57:48.230880 10357 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:57:48.237962 10357 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:57:48.238391 10357 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:57:48.239591 10357 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:57:48.241330 10357 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:57:48.241418 10357 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:57:48.241600 10357 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:57:48.241662 10357 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:57:48.278412 10357 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.1:41971
I20260807 22:57:48.278551 10601 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.1:41971 every 8 connection(s)
I20260807 22:57:48.279860 10357 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786143361779964-25976-0/minicluster-data/ts-0/data/info.pb
I20260807 22:57:48.285598 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 10357
I20260807 22:57:48.285853 25976 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786143361779964-25976-0/minicluster-data/ts-0/wal/instance
I20260807 22:57:48.286693 25976 external_mini_cluster.cc:1428] Setting key 1faf8829fc83459395521181a79dcf73
Aug 07 22:57:48 dist-test-slave-z8x9 krb5kdc[10160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143468, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.1@KRBTEST.COM for kudu/127.25.94.62@KRBTEST.COM
I20260807 22:57:48.304101 10607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:48.282729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.1:50219 (local address 127.25.94.62:35335)
0807 22:57:48.283172 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:48.283178 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:48.284133 (+ 955us) server_negotiation.cc:407] Connection header received
0807 22:57:48.286554 (+ 2421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:48.286559 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:48.286633 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:48.286752 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:48.288743 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:48.289841 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:48.291439 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:48.292183 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:48.292342 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:48.297292 (+ 4950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:57:48.297298 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:57:48.297301 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:57:48.297337 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:57:48.300181 (+ 2844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:48.301356 (+ 1175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:48.301361 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:48.301364 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:48.301519 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:48.302097 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:48.302101 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:48.302103 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:48.302418 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:57:48.302548 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0807 22:57:48.303581 (+ 1033us) server_negotiation.cc:299] Negotiation successful
0807 22:57:48.303794 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":228,"threads_started":1}
I20260807 22:57:48.305830 10608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:48.283242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:35335 (local address 127.25.94.1:50219)
0807 22:57:48.283970 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:48.284010 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0807 22:57:48.286182 (+ 2172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:57:48.287048 (+ 866us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:57:48.287060 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:57:48.287828 (+ 768us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:57:48.288489 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:48.288519 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:48.290003 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:48.290007 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:48.291247 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:48.291260 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:48.292448 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:48.292455 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:48.292724 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:48.293600 (+ 876us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:57:48.293637 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:57:48.296851 (+ 3214us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:57:48.300372 (+ 3521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:57:48.300382 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:57:48.300408 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:57:48.301120 (+ 712us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:57:48.301671 (+ 551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:57:48.301679 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:57:48.301685 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:57:48.301973 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:57:48.302563 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:57:48.302575 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:57:48.303111 (+ 536us) client_negotiation.cc:769] Sending connection context
0807 22:57:48.303468 (+ 357us) client_negotiation.cc:240] Negotiation successful
0807 22:57:48.304105 (+ 637us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":271,"threads_started":1}
I20260807 22:57:48.307565 10604 heartbeater.cc:347] Connected to a master server at 127.25.94.62:35335
I20260807 22:57:48.307930 10604 heartbeater.cc:464] Registering TS with master...
I20260807 22:57:48.308856 10604 heartbeater.cc:511] Master 127.25.94.62:35335 requested a full tablet report, sending...
I20260807 22:57:48.311211 10242 ts_manager.cc:194] Registered new tserver with Master: 28bb0f3e219e46489ba58f36a2f92f96 (127.25.94.1:41971)
I20260807 22:57:48.312841 10242 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.94.1@KRBTEST.COM'} at 127.25.94.1:50219
WARNING: no policy specified for kudu/127.25.94.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.2.keytab.
Entry for principal kudu/127.25.94.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.2.keytab.
I20260807 22:57:48.353641 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786143361779964-25976-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786143361779964-25976-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786143361779964-25976-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786143361779964-25976-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.2:0
--local_ip_for_outbound_sockets=127.25.94.2
--webserver_interface=127.25.94.2
--webserver_port=0
--tserver_master_addrs=127.25.94.62:35335
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.2.keytab
--principal=kudu/127.25.94.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.20:38243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786143361779964-25976-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786143361779964-25976-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786143361779964-25976-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:57:48.698086 10612 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:57:48.698450 10612 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:57:48.698618 10612 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:57:48.706101 10612 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:57:48.706250 10612 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:57:48.706429 10612 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.2
I20260807 22:57:48.716663 10612 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:38243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786143361779964-25976-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786143361779964-25976-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.2.keytab
--principal=kudu/127.25.94.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786143361779964-25976-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.94.2
--webserver_port=0
--tserver_master_addrs=127.25.94.62:35335
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.2
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786143361779964-25976-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 22:57:48.718077 10612 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:57:48.719360 10612 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:57:48.732770 10619 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:57:48.732708 10618 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:57:48.733162 10621 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:57:48.733901 10612 server_base.cc:1034] running on GCE node
I20260807 22:57:48.734539 10612 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:57:48.735567 10612 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:57:48.736792 10612 hybrid_clock.cc:648] HybridClock initialized: now 1786143468736751 us; error 55 us; skew 500 ppm
I20260807 22:57:48.737105 10612 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 22:57:48 dist-test-slave-z8x9 krb5kdc[10160](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143468, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:57:48.740430 10612 init.cc:377] Logged in from keytab as kudu/127.25.94.2@KRBTEST.COM (short username kudu)
I20260807 22:57:48.741919 10612 webserver.cc:460] Webserver started at http://127.25.94.2:35763/ using document root <none> and password file <none>
I20260807 22:57:48.742662 10612 fs_manager.cc:359] Metadata directory not provided
I20260807 22:57:48.742805 10612 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:57:48.743124 10612 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:57:48.746101 10612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786143361779964-25976-0/minicluster-data/ts-1/data/instance:
uuid: "bf1ab86fe5b74ecd874f982def0b3030"
format_stamp: "Formatted at 2026-08-07 22:57:48 on dist-test-slave-z8x9"
server_key: "a549e254e9d625a215dcc414ba1fa1aa"
server_key_iv: "251d3712721355db45751e40232ec484"
server_key_version: "encryptionkey@0"
I20260807 22:57:48.746886 10612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786143361779964-25976-0/minicluster-data/ts-1/wal/instance:
uuid: "bf1ab86fe5b74ecd874f982def0b3030"
format_stamp: "Formatted at 2026-08-07 22:57:48 on dist-test-slave-z8x9"
server_key: "a549e254e9d625a215dcc414ba1fa1aa"
server_key_iv: "251d3712721355db45751e40232ec484"
server_key_version: "encryptionkey@0"
I20260807 22:57:48.752328 10612 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260807 22:57:48.755867 10631 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:57:48.757306 10612 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 22:57:48.757473 10612 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786143361779964-25976-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786143361779964-25976-0/minicluster-data/ts-1/wal
uuid: "bf1ab86fe5b74ecd874f982def0b3030"
format_stamp: "Formatted at 2026-08-07 22:57:48 on dist-test-slave-z8x9"
server_key: "a549e254e9d625a215dcc414ba1fa1aa"
server_key_iv: "251d3712721355db45751e40232ec484"
server_key_version: "encryptionkey@0"
I20260807 22:57:48.757684 10612 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786143361779964-25976-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786143361779964-25976-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786143361779964-25976-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:57:48.807349 10612 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:57:48.812086 10612 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:57:48.812407 10612 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:57:48.813478 10612 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:57:48.814937 10612 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:57:48.815042 10612 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:57:48.815202 10612 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:57:48.815254 10612 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:57:48.850425 10612 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.2:37595
I20260807 22:57:48.850519 10819 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.2:37595 every 8 connection(s)
I20260807 22:57:48.852075 10612 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786143361779964-25976-0/minicluster-data/ts-1/data/info.pb
I20260807 22:57:48.862298 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 10612
I20260807 22:57:48.862515 25976 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786143361779964-25976-0/minicluster-data/ts-1/wal/instance
I20260807 22:57:48.862977 25976 external_mini_cluster.cc:1428] Setting key 8f63c87ec3fc0f883ff6ee3e90358b80
Aug 07 22:57:48 dist-test-slave-z8x9 krb5kdc[10160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143468, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.2@KRBTEST.COM for kudu/127.25.94.62@KRBTEST.COM
I20260807 22:57:48.873762 10825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:48.855227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.2:53921 (local address 127.25.94.62:35335)
0807 22:57:48.855661 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:48.855665 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:48.856693 (+ 1028us) server_negotiation.cc:407] Connection header received
0807 22:57:48.858624 (+ 1931us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:48.858630 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:48.858726 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:48.858857 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:48.860372 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:48.861231 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:48.862990 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:48.863477 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:48.863630 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:48.867821 (+ 4191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:57:48.867827 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:57:48.867830 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:57:48.867862 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:57:48.870645 (+ 2783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:48.871391 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:48.871399 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:48.871405 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:48.871539 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:48.871975 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:48.871981 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:48.871984 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:48.872261 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:57:48.872419 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0807 22:57:48.873107 (+ 688us) server_negotiation.cc:299] Negotiation successful
0807 22:57:48.873361 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":210,"threads_started":1}
I20260807 22:57:48.874716 10826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:48.855833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:35335 (local address 127.25.94.2:53921)
0807 22:57:48.856531 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:48.856574 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0807 22:57:48.858342 (+ 1768us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:57:48.859088 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:57:48.859097 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:57:48.859631 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:57:48.860194 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:48.860211 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:48.861382 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:48.861390 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:48.862848 (+ 1458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:48.862861 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:48.863769 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:48.863779 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:48.864078 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:48.864867 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:57:48.864905 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:57:48.867567 (+ 2662us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:57:48.870802 (+ 3235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:57:48.870810 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:57:48.870829 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:57:48.871265 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:57:48.871654 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:57:48.871659 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:57:48.871662 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:57:48.871857 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:57:48.872392 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:57:48.872400 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:57:48.872746 (+ 346us) client_negotiation.cc:769] Sending connection context
0807 22:57:48.873018 (+ 272us) client_negotiation.cc:240] Negotiation successful
0807 22:57:48.873601 (+ 583us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":280,"threads_started":1}
I20260807 22:57:48.876240 10822 heartbeater.cc:347] Connected to a master server at 127.25.94.62:35335
I20260807 22:57:48.876575 10822 heartbeater.cc:464] Registering TS with master...
I20260807 22:57:48.877555 10822 heartbeater.cc:511] Master 127.25.94.62:35335 requested a full tablet report, sending...
I20260807 22:57:48.879297 10242 ts_manager.cc:194] Registered new tserver with Master: bf1ab86fe5b74ecd874f982def0b3030 (127.25.94.2:37595)
I20260807 22:57:48.880626 10242 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.94.2@KRBTEST.COM'} at 127.25.94.2:53921
WARNING: no policy specified for kudu/127.25.94.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.3.keytab.
Entry for principal kudu/127.25.94.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.3.keytab.
I20260807 22:57:48.925518 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786143361779964-25976-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786143361779964-25976-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786143361779964-25976-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786143361779964-25976-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.3:0
--local_ip_for_outbound_sockets=127.25.94.3
--webserver_interface=127.25.94.3
--webserver_port=0
--tserver_master_addrs=127.25.94.62:35335
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.3.keytab
--principal=kudu/127.25.94.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.20:38243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786143361779964-25976-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786143361779964-25976-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786143361779964-25976-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:57:49.269970 10830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:57:49.270270 10830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:57:49.270375 10830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:57:49.277902 10830 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:57:49.278028 10830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:57:49.278242 10830 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.3
I20260807 22:57:49.288362 10830 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:38243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786143361779964-25976-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786143361779964-25976-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.3.keytab
--principal=kudu/127.25.94.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786143361779964-25976-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.94.3
--webserver_port=0
--tserver_master_addrs=127.25.94.62:35335
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.3
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786143361779964-25976-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 22:57:49.289822 10830 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:57:49.291195 10830 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:57:49.305449 10839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:57:49.305480 10843 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:57:49.305521 10840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:57:49.306757 10830 server_base.cc:1034] running on GCE node
I20260807 22:57:49.307471 10830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:57:49.309013 10830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:57:49.310344 10830 hybrid_clock.cc:648] HybridClock initialized: now 1786143469310303 us; error 84 us; skew 500 ppm
I20260807 22:57:49.310755 10830 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 22:57:49 dist-test-slave-z8x9 krb5kdc[10160](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143469, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:57:49.314455 10830 init.cc:377] Logged in from keytab as kudu/127.25.94.3@KRBTEST.COM (short username kudu)
I20260807 22:57:49.316124 10830 webserver.cc:460] Webserver started at http://127.25.94.3:38207/ using document root <none> and password file <none>
I20260807 22:57:49.316151 10604 heartbeater.cc:503] Master 127.25.94.62:35335 was elected leader, sending a full tablet report...
I20260807 22:57:49.317031 10830 fs_manager.cc:359] Metadata directory not provided
I20260807 22:57:49.317236 10830 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:57:49.317584 10830 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:57:49.320753 10830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786143361779964-25976-0/minicluster-data/ts-2/data/instance:
uuid: "50d15ab368f242d6ba9acc047b4de0c7"
format_stamp: "Formatted at 2026-08-07 22:57:49 on dist-test-slave-z8x9"
server_key: "faaffaa1b9265f247c3c43371c384389"
server_key_iv: "d53dd9862af5fea5a7ec59e14a10dcbd"
server_key_version: "encryptionkey@0"
I20260807 22:57:49.321620 10830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786143361779964-25976-0/minicluster-data/ts-2/wal/instance:
uuid: "50d15ab368f242d6ba9acc047b4de0c7"
format_stamp: "Formatted at 2026-08-07 22:57:49 on dist-test-slave-z8x9"
server_key: "faaffaa1b9265f247c3c43371c384389"
server_key_iv: "d53dd9862af5fea5a7ec59e14a10dcbd"
server_key_version: "encryptionkey@0"
I20260807 22:57:49.327157 10830 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260807 22:57:49.330978 10855 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:57:49.332581 10830 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260807 22:57:49.332783 10830 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786143361779964-25976-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786143361779964-25976-0/minicluster-data/ts-2/wal
uuid: "50d15ab368f242d6ba9acc047b4de0c7"
format_stamp: "Formatted at 2026-08-07 22:57:49 on dist-test-slave-z8x9"
server_key: "faaffaa1b9265f247c3c43371c384389"
server_key_iv: "d53dd9862af5fea5a7ec59e14a10dcbd"
server_key_version: "encryptionkey@0"
I20260807 22:57:49.333006 10830 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786143361779964-25976-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786143361779964-25976-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786143361779964-25976-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:57:49.371330 10830 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:57:49.376447 10830 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:57:49.376762 10830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:57:49.377856 10830 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:57:49.379328 10830 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:57:49.379411 10830 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:57:49.379562 10830 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:57:49.379617 10830 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:57:49.415446 10830 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.3:40037
I20260807 22:57:49.415567 11046 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.3:40037 every 8 connection(s)
I20260807 22:57:49.416774 10830 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786143361779964-25976-0/minicluster-data/ts-2/data/info.pb
I20260807 22:57:49.423417 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 10830
I20260807 22:57:49.423630 25976 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786143361779964-25976-0/minicluster-data/ts-2/wal/instance
I20260807 22:57:49.424077 25976 external_mini_cluster.cc:1428] Setting key d085d08b930c750e5616691d361269a3
Aug 07 22:57:49 dist-test-slave-z8x9 krb5kdc[10160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143469, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.3@KRBTEST.COM for kudu/127.25.94.62@KRBTEST.COM
I20260807 22:57:49.439952 11049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:49.419418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.3:49129 (local address 127.25.94.62:35335)
0807 22:57:49.419896 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:49.419901 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:49.420675 (+ 774us) server_negotiation.cc:407] Connection header received
0807 22:57:49.422531 (+ 1856us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:49.422537 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:49.422630 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:49.422750 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:49.424419 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:49.425476 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:49.427627 (+ 2151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:49.428112 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:49.428219 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:49.432804 (+ 4585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:57:49.432810 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:57:49.432814 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:57:49.432860 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:57:49.436096 (+ 3236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:49.437048 (+ 952us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:49.437053 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:49.437055 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:49.437282 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:49.437719 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:49.437723 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:49.437726 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:49.438152 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:57:49.438268 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0807 22:57:49.439349 (+ 1081us) server_negotiation.cc:299] Negotiation successful
0807 22:57:49.439568 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":250,"threads_started":1}
I20260807 22:57:49.441246 11051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:49.419816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:35335 (local address 127.25.94.3:49129)
0807 22:57:49.420519 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:49.420562 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0807 22:57:49.422250 (+ 1688us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:57:49.422991 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:57:49.422999 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:57:49.423573 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:57:49.424245 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:49.424264 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:49.425732 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:49.425739 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:49.427434 (+ 1695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:49.427455 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:49.428348 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:49.428354 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:49.428585 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:49.429364 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:57:49.429416 (+ 52us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:57:49.432441 (+ 3025us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:57:49.436311 (+ 3870us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:57:49.436322 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:57:49.436342 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:57:49.436905 (+ 563us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:57:49.437383 (+ 478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:57:49.437386 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:57:49.437388 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:57:49.437584 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:57:49.438365 (+ 781us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:57:49.438374 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:57:49.438867 (+ 493us) client_negotiation.cc:769] Sending connection context
0807 22:57:49.439215 (+ 348us) client_negotiation.cc:240] Negotiation successful
0807 22:57:49.439830 (+ 615us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":286,"threads_started":1}
I20260807 22:57:49.442535 11047 heartbeater.cc:347] Connected to a master server at 127.25.94.62:35335
I20260807 22:57:49.442874 11047 heartbeater.cc:464] Registering TS with master...
I20260807 22:57:49.443636 11047 heartbeater.cc:511] Master 127.25.94.62:35335 requested a full tablet report, sending...
I20260807 22:57:49.445253 10242 ts_manager.cc:194] Registered new tserver with Master: 50d15ab368f242d6ba9acc047b4de0c7 (127.25.94.3:40037)
I20260807 22:57:49.446079 10242 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.94.3@KRBTEST.COM'} at 127.25.94.3:49129
I20260807 22:57:49.452852 25976 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 22:57:49.471696 11049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:49.457905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52710 (local address 127.25.94.62:35335)
0807 22:57:49.458088 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:49.458093 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:49.458598 (+ 505us) server_negotiation.cc:407] Connection header received
0807 22:57:49.458742 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:49.458747 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:49.458826 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:49.458904 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:49.460349 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:49.461265 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:49.462775 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:49.463190 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:49.463312 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:49.466178 (+ 2866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:57:49.466184 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:57:49.466187 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:57:49.466261 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:57:49.469240 (+ 2979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:49.469926 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:49.469930 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:49.469932 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:49.470054 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:49.470386 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:49.470390 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:49.470392 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:49.470740 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:57:49.470939 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0807 22:57:49.471265 (+ 326us) server_negotiation.cc:299] Negotiation successful
0807 22:57:49.471448 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260807 22:57:49.476511 10242 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52710:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 22:57:49.484176 10242 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 22:57:49.510553 11079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:49.500284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43848 (local address 127.25.94.3:40037)
0807 22:57:49.500929 (+ 645us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:49.500938 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:49.502239 (+ 1301us) server_negotiation.cc:407] Connection header received
0807 22:57:49.502359 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:49.502364 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:49.502507 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:49.502709 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:57:49.503403 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:49.505115 (+ 1712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:49.507916 (+ 2801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:49.509368 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:49.509497 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:49.509755 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0807 22:57:49.509985 (+ 230us) server_negotiation.cc:299] Negotiation successful
0807 22:57:49.510148 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":489,"thread_start_us":323,"threads_started":1}
I20260807 22:57:49.510696 11081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:49.499781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.3:40037 (local address 127.0.0.1:43848)
0807 22:57:49.501995 (+ 2214us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:49.502014 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 22:57:49.502160 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:57:49.502700 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:57:49.502704 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:57:49.502729 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:57:49.503185 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:49.503199 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:49.505397 (+ 2198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:49.505404 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:49.507750 (+ 2346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:49.507766 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:49.509579 (+ 1813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:49.509589 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:49.509769 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:49.509800 (+ 31us) client_negotiation.cc:769] Sending connection context
0807 22:57:49.509912 (+ 112us) client_negotiation.cc:240] Negotiation successful
0807 22:57:49.510148 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2059,"mutex_wait_us":2,"spinlock_wait_cycles":896,"thread_start_us":1200,"threads_started":1}
I20260807 22:57:49.510819 11070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:49.497518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.2:37595 (local address 127.0.0.1:44808)
0807 22:57:49.500748 (+ 3230us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:49.500769 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 22:57:49.500994 (+ 225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:57:49.501852 (+ 858us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:57:49.501857 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:57:49.501883 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:57:49.503286 (+ 1403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:49.503299 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:49.505732 (+ 2433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:49.505737 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:49.507984 (+ 2247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:49.507994 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:49.509924 (+ 1930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:49.509930 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:49.510098 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:49.510127 (+ 29us) client_negotiation.cc:769] Sending connection context
0807 22:57:49.510212 (+ 85us) client_negotiation.cc:240] Negotiation successful
0807 22:57:49.510347 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3095,"mutex_wait_us":784,"spinlock_wait_cycles":9984,"thread_start_us":3274,"threads_started":1}
I20260807 22:57:49.511539 11073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:49.498292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44808 (local address 127.25.94.2:37595)
0807 22:57:49.500391 (+ 2099us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:49.500401 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:49.500871 (+ 470us) server_negotiation.cc:407] Connection header received
0807 22:57:49.501243 (+ 372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:49.501254 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:49.501481 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:49.501680 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:57:49.503434 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:49.505461 (+ 2027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:49.508131 (+ 2670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:49.509520 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:49.509646 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:49.509857 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0807 22:57:49.511015 (+ 1158us) server_negotiation.cc:299] Negotiation successful
0807 22:57:49.511188 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1934,"thread_start_us":1827,"threads_started":1}
I20260807 22:57:49.510751 11069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:49.497284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.1:41971 (local address 127.0.0.1:52718)
0807 22:57:49.498766 (+ 1482us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:49.498817 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0807 22:57:49.499017 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:57:49.501373 (+ 2356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:57:49.501380 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:57:49.501412 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:57:49.502281 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:49.502299 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:49.503811 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:49.503816 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:49.507240 (+ 3424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:49.507252 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:49.509466 (+ 2214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:49.509472 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:49.509680 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:49.509739 (+ 59us) client_negotiation.cc:769] Sending connection context
0807 22:57:49.509889 (+ 150us) client_negotiation.cc:240] Negotiation successful
0807 22:57:49.510151 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1123,"mutex_wait_us":122,"thread_start_us":914,"threads_started":1}
I20260807 22:57:49.511705 11071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:49.497641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52718 (local address 127.25.94.1:41971)
0807 22:57:49.499718 (+ 2077us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:49.499728 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:49.499756 (+ 28us) server_negotiation.cc:407] Connection header received
0807 22:57:49.499857 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:49.499863 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:49.501094 (+ 1231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:49.501967 (+ 873us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:57:49.502481 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:49.503660 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:49.507422 (+ 3762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:49.509220 (+ 1798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:49.510871 (+ 1651us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:49.511107 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0807 22:57:49.511186 (+ 79us) server_negotiation.cc:299] Negotiation successful
0807 22:57:49.511355 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1874,"thread_start_us":1702,"threads_started":1}
I20260807 22:57:49.515100 10933 tablet_service.cc:1467] Processing CreateTablet for tablet ae32fd67806c4c82b19c11b9e5657b64 (DEFAULT_TABLE table=test-table [id=a43b8e88d508469a9068be2caa4959b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:57:49.515257 10709 tablet_service.cc:1467] Processing CreateTablet for tablet ae32fd67806c4c82b19c11b9e5657b64 (DEFAULT_TABLE table=test-table [id=a43b8e88d508469a9068be2caa4959b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:57:49.515259 10478 tablet_service.cc:1467] Processing CreateTablet for tablet ae32fd67806c4c82b19c11b9e5657b64 (DEFAULT_TABLE table=test-table [id=a43b8e88d508469a9068be2caa4959b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:57:49.516482 10933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae32fd67806c4c82b19c11b9e5657b64. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:57:49.516609 10478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae32fd67806c4c82b19c11b9e5657b64. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:57:49.516607 10709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae32fd67806c4c82b19c11b9e5657b64. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:57:49.529242 11082 tablet_bootstrap.cc:492] T ae32fd67806c4c82b19c11b9e5657b64 P 50d15ab368f242d6ba9acc047b4de0c7: Bootstrap starting.
I20260807 22:57:49.531209 11083 tablet_bootstrap.cc:492] T ae32fd67806c4c82b19c11b9e5657b64 P bf1ab86fe5b74ecd874f982def0b3030: Bootstrap starting.
I20260807 22:57:49.533268 11082 tablet_bootstrap.cc:654] T ae32fd67806c4c82b19c11b9e5657b64 P 50d15ab368f242d6ba9acc047b4de0c7: Neither blocks nor log segments found. Creating new log.
I20260807 22:57:49.534840 11082 log.cc:824] T ae32fd67806c4c82b19c11b9e5657b64 P 50d15ab368f242d6ba9acc047b4de0c7: Log is configured to *not* fsync() on all Append() calls
I20260807 22:57:49.534914 11083 tablet_bootstrap.cc:654] T ae32fd67806c4c82b19c11b9e5657b64 P bf1ab86fe5b74ecd874f982def0b3030: Neither blocks nor log segments found. Creating new log.
I20260807 22:57:49.535823 11084 tablet_bootstrap.cc:492] T ae32fd67806c4c82b19c11b9e5657b64 P 28bb0f3e219e46489ba58f36a2f92f96: Bootstrap starting.
I20260807 22:57:49.536182 11083 log.cc:824] T ae32fd67806c4c82b19c11b9e5657b64 P bf1ab86fe5b74ecd874f982def0b3030: Log is configured to *not* fsync() on all Append() calls
I20260807 22:57:49.539563 11083 tablet_bootstrap.cc:492] T ae32fd67806c4c82b19c11b9e5657b64 P bf1ab86fe5b74ecd874f982def0b3030: No bootstrap required, opened a new log
I20260807 22:57:49.539613 11084 tablet_bootstrap.cc:654] T ae32fd67806c4c82b19c11b9e5657b64 P 28bb0f3e219e46489ba58f36a2f92f96: Neither blocks nor log segments found. Creating new log.
I20260807 22:57:49.539871 11083 ts_tablet_manager.cc:1397] T ae32fd67806c4c82b19c11b9e5657b64 P bf1ab86fe5b74ecd874f982def0b3030: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260807 22:57:49.540123 11082 tablet_bootstrap.cc:492] T ae32fd67806c4c82b19c11b9e5657b64 P 50d15ab368f242d6ba9acc047b4de0c7: No bootstrap required, opened a new log
I20260807 22:57:49.540405 11082 ts_tablet_manager.cc:1397] T ae32fd67806c4c82b19c11b9e5657b64 P 50d15ab368f242d6ba9acc047b4de0c7: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.003s
I20260807 22:57:49.540889 11084 log.cc:824] T ae32fd67806c4c82b19c11b9e5657b64 P 28bb0f3e219e46489ba58f36a2f92f96: Log is configured to *not* fsync() on all Append() calls
I20260807 22:57:49.544173 11084 tablet_bootstrap.cc:492] T ae32fd67806c4c82b19c11b9e5657b64 P 28bb0f3e219e46489ba58f36a2f92f96: No bootstrap required, opened a new log
I20260807 22:57:49.544440 11084 ts_tablet_manager.cc:1397] T ae32fd67806c4c82b19c11b9e5657b64 P 28bb0f3e219e46489ba58f36a2f92f96: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260807 22:57:49.548188 11083 raft_consensus.cc:348] T ae32fd67806c4c82b19c11b9e5657b64 P bf1ab86fe5b74ecd874f982def0b3030 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28bb0f3e219e46489ba58f36a2f92f96" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 41971 } } peers { permanent_uuid: "bf1ab86fe5b74ecd874f982def0b3030" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 37595 } } peers { permanent_uuid: "50d15ab368f242d6ba9acc047b4de0c7" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 40037 } }
I20260807 22:57:49.548575 11083 raft_consensus.cc:374] T ae32fd67806c4c82b19c11b9e5657b64 P bf1ab86fe5b74ecd874f982def0b3030 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:57:49.548692 11083 raft_consensus.cc:729] T ae32fd67806c4c82b19c11b9e5657b64 P bf1ab86fe5b74ecd874f982def0b3030 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf1ab86fe5b74ecd874f982def0b3030, State: Initialized, Role: FOLLOWER
I20260807 22:57:49.549387 11083 consensus_queue.cc:260] T ae32fd67806c4c82b19c11b9e5657b64 P bf1ab86fe5b74ecd874f982def0b3030 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28bb0f3e219e46489ba58f36a2f92f96" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 41971 } } peers { permanent_uuid: "bf1ab86fe5b74ecd874f982def0b3030" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 37595 } } peers { permanent_uuid: "50d15ab368f242d6ba9acc047b4de0c7" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 40037 } }
I20260807 22:57:49.550547 10822 heartbeater.cc:503] Master 127.25.94.62:35335 was elected leader, sending a full tablet report...
I20260807 22:57:49.550809 11082 raft_consensus.cc:348] T ae32fd67806c4c82b19c11b9e5657b64 P 50d15ab368f242d6ba9acc047b4de0c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28bb0f3e219e46489ba58f36a2f92f96" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 41971 } } peers { permanent_uuid: "bf1ab86fe5b74ecd874f982def0b3030" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 37595 } } peers { permanent_uuid: "50d15ab368f242d6ba9acc047b4de0c7" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 40037 } }
I20260807 22:57:49.551263 11082 raft_consensus.cc:374] T ae32fd67806c4c82b19c11b9e5657b64 P 50d15ab368f242d6ba9acc047b4de0c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:57:49.551337 11083 ts_tablet_manager.cc:1428] T ae32fd67806c4c82b19c11b9e5657b64 P bf1ab86fe5b74ecd874f982def0b3030: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260807 22:57:49.551362 11082 raft_consensus.cc:729] T ae32fd67806c4c82b19c11b9e5657b64 P 50d15ab368f242d6ba9acc047b4de0c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50d15ab368f242d6ba9acc047b4de0c7, State: Initialized, Role: FOLLOWER
I20260807 22:57:49.551924 11082 consensus_queue.cc:260] T ae32fd67806c4c82b19c11b9e5657b64 P 50d15ab368f242d6ba9acc047b4de0c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28bb0f3e219e46489ba58f36a2f92f96" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 41971 } } peers { permanent_uuid: "bf1ab86fe5b74ecd874f982def0b3030" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 37595 } } peers { permanent_uuid: "50d15ab368f242d6ba9acc047b4de0c7" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 40037 } }
I20260807 22:57:49.553089 11047 heartbeater.cc:503] Master 127.25.94.62:35335 was elected leader, sending a full tablet report...
I20260807 22:57:49.553700 11082 ts_tablet_manager.cc:1428] T ae32fd67806c4c82b19c11b9e5657b64 P 50d15ab368f242d6ba9acc047b4de0c7: Time spent starting tablet: real 0.013s user 0.010s sys 0.004s
I20260807 22:57:49.555189 11084 raft_consensus.cc:348] T ae32fd67806c4c82b19c11b9e5657b64 P 28bb0f3e219e46489ba58f36a2f92f96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28bb0f3e219e46489ba58f36a2f92f96" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 41971 } } peers { permanent_uuid: "bf1ab86fe5b74ecd874f982def0b3030" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 37595 } } peers { permanent_uuid: "50d15ab368f242d6ba9acc047b4de0c7" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 40037 } }
I20260807 22:57:49.555655 11084 raft_consensus.cc:374] T ae32fd67806c4c82b19c11b9e5657b64 P 28bb0f3e219e46489ba58f36a2f92f96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:57:49.555795 11084 raft_consensus.cc:729] T ae32fd67806c4c82b19c11b9e5657b64 P 28bb0f3e219e46489ba58f36a2f92f96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28bb0f3e219e46489ba58f36a2f92f96, State: Initialized, Role: FOLLOWER
I20260807 22:57:49.556401 11084 consensus_queue.cc:260] T ae32fd67806c4c82b19c11b9e5657b64 P 28bb0f3e219e46489ba58f36a2f92f96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28bb0f3e219e46489ba58f36a2f92f96" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 41971 } } peers { permanent_uuid: "bf1ab86fe5b74ecd874f982def0b3030" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 37595 } } peers { permanent_uuid: "50d15ab368f242d6ba9acc047b4de0c7" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 40037 } }
I20260807 22:57:49.558144 11084 ts_tablet_manager.cc:1428] T ae32fd67806c4c82b19c11b9e5657b64 P 28bb0f3e219e46489ba58f36a2f92f96: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
W20260807 22:57:49.604638 10824 tablet.cc:2369] T ae32fd67806c4c82b19c11b9e5657b64 P bf1ab86fe5b74ecd874f982def0b3030: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 22:57:49.630962 11090 raft_consensus.cc:482] T ae32fd67806c4c82b19c11b9e5657b64 P bf1ab86fe5b74ecd874f982def0b3030 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 22:57:49.631289 11090 raft_consensus.cc:504] T ae32fd67806c4c82b19c11b9e5657b64 P bf1ab86fe5b74ecd874f982def0b3030 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28bb0f3e219e46489ba58f36a2f92f96" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 41971 } } peers { permanent_uuid: "bf1ab86fe5b74ecd874f982def0b3030" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 37595 } } peers { permanent_uuid: "50d15ab368f242d6ba9acc047b4de0c7" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 40037 } }
I20260807 22:57:49.633378 11090 leader_election.cc:288] T ae32fd67806c4c82b19c11b9e5657b64 P bf1ab86fe5b74ecd874f982def0b3030 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28bb0f3e219e46489ba58f36a2f92f96 (127.25.94.1:41971), 50d15ab368f242d6ba9acc047b4de0c7 (127.25.94.3:40037)
I20260807 22:57:49.640825 11079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:49.633941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.2:48663 (local address 127.25.94.3:40037)
0807 22:57:49.634162 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:49.634168 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:49.635174 (+ 1006us) server_negotiation.cc:407] Connection header received
0807 22:57:49.635368 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:49.635372 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:49.635516 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:49.635637 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:57:49.636324 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:49.637137 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:49.639042 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:49.639795 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:49.639893 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:49.640017 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0807 22:57:49.640454 (+ 437us) server_negotiation.cc:299] Negotiation successful
0807 22:57:49.640577 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260807 22:57:49.640825 11071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:49.633874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.2:60577 (local address 127.25.94.1:41971)
0807 22:57:49.634112 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:49.634118 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:49.635083 (+ 965us) server_negotiation.cc:407] Connection header received
0807 22:57:49.635295 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:49.635304 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:49.635384 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:49.635454 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:57:49.636199 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:49.637032 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:49.639082 (+ 2050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:49.639875 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:49.640029 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:49.640129 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0807 22:57:49.640430 (+ 301us) server_negotiation.cc:299] Negotiation successful
0807 22:57:49.640564 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260807 22:57:49.641009 11094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:49.634081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.1:41971 (local address 127.25.94.2:60577)
0807 22:57:49.634958 (+ 877us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:49.634975 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 22:57:49.635106 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:57:49.635564 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:57:49.635572 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:57:49.635608 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:57:49.636008 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:49.636018 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:49.637209 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:49.637214 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:49.638895 (+ 1681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:49.638907 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:49.640129 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:49.640134 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:49.640242 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:49.640286 (+ 44us) client_negotiation.cc:769] Sending connection context
0807 22:57:49.640363 (+ 77us) client_negotiation.cc:240] Negotiation successful
0807 22:57:49.640525 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":737,"mutex_wait_us":109,"thread_start_us":480,"threads_started":1}
I20260807 22:57:49.641039 11095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:49.634143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.3:40037 (local address 127.25.94.2:48663)
0807 22:57:49.635021 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0807 22:57:49.635061 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0807 22:57:49.635228 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:57:49.635674 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:57:49.635677 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:57:49.635735 (+ 58us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:57:49.636150 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:49.636164 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:49.637343 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:49.637349 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:49.638892 (+ 1543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:57:49.638905 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:57:49.640036 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:57:49.640042 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:57:49.640240 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:49.640296 (+ 56us) client_negotiation.cc:769] Sending connection context
0807 22:57:49.640405 (+ 109us) client_negotiation.cc:240] Negotiation successful
0807 22:57:49.640546 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":774,"mutex_wait_us":31,"spinlock_wait_cycles":1280,"thread_start_us":499,"threads_started":1}
I20260807 22:57:49.642419 10967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae32fd67806c4c82b19c11b9e5657b64" candidate_uuid: "bf1ab86fe5b74ecd874f982def0b3030" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50d15ab368f242d6ba9acc047b4de0c7" is_pre_election: true
I20260807 22:57:49.642413 10517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae32fd67806c4c82b19c11b9e5657b64" candidate_uuid: "bf1ab86fe5b74ecd874f982def0b3030" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28bb0f3e219e46489ba58f36a2f92f96" is_pre_election: true
I20260807 22:57:49.642948 10517 raft_consensus.cc:2454] T ae32fd67806c4c82b19c11b9e5657b64 P 28bb0f3e219e46489ba58f36a2f92f96 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf1ab86fe5b74ecd874f982def0b3030 in term 0.
I20260807 22:57:49.642949 10967 raft_consensus.cc:2454] T ae32fd67806c4c82b19c11b9e5657b64 P 50d15ab368f242d6ba9acc047b4de0c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf1ab86fe5b74ecd874f982def0b3030 in term 0.
I20260807 22:57:49.643689 10640 leader_election.cc:302] T ae32fd67806c4c82b19c11b9e5657b64 P bf1ab86fe5b74ecd874f982def0b3030 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 28bb0f3e219e46489ba58f36a2f92f96, 50d15ab368f242d6ba9acc047b4de0c7, bf1ab86fe5b74ecd874f982def0b3030; no voters:
I20260807 22:57:49.644146 11090 raft_consensus.cc:2781] T ae32fd67806c4c82b19c11b9e5657b64 P bf1ab86fe5b74ecd874f982def0b3030 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 22:57:49.644311 11090 raft_consensus.cc:482] T ae32fd67806c4c82b19c11b9e5657b64 P bf1ab86fe5b74ecd874f982def0b3030 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 22:57:49.644421 11090 raft_consensus.cc:3037] T ae32fd67806c4c82b19c11b9e5657b64 P bf1ab86fe5b74ecd874f982def0b3030 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:57:49.647364 11090 raft_consensus.cc:504] T ae32fd67806c4c82b19c11b9e5657b64 P bf1ab86fe5b74ecd874f982def0b3030 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28bb0f3e219e46489ba58f36a2f92f96" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 41971 } } peers { permanent_uuid: "bf1ab86fe5b74ecd874f982def0b3030" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 37595 } } peers { permanent_uuid: "50d15ab368f242d6ba9acc047b4de0c7" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 40037 } }
I20260807 22:57:49.648034 11090 leader_election.cc:288] T ae32fd67806c4c82b19c11b9e5657b64 P bf1ab86fe5b74ecd874f982def0b3030 [CANDIDATE]: Term 1 election: Requested vote from peers 28bb0f3e219e46489ba58f36a2f92f96 (127.25.94.1:41971), 50d15ab368f242d6ba9acc047b4de0c7 (127.25.94.3:40037)
I20260807 22:57:49.648562 10967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae32fd67806c4c82b19c11b9e5657b64" candidate_uuid: "bf1ab86fe5b74ecd874f982def0b3030" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50d15ab368f242d6ba9acc047b4de0c7"
I20260807 22:57:49.648563 10517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae32fd67806c4c82b19c11b9e5657b64" candidate_uuid: "bf1ab86fe5b74ecd874f982def0b3030" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28bb0f3e219e46489ba58f36a2f92f96"
I20260807 22:57:49.648762 10967 raft_consensus.cc:3037] T ae32fd67806c4c82b19c11b9e5657b64 P 50d15ab368f242d6ba9acc047b4de0c7 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:57:49.648782 10517 raft_consensus.cc:3037] T ae32fd67806c4c82b19c11b9e5657b64 P 28bb0f3e219e46489ba58f36a2f92f96 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:57:49.651017 10517 raft_consensus.cc:2454] T ae32fd67806c4c82b19c11b9e5657b64 P 28bb0f3e219e46489ba58f36a2f92f96 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf1ab86fe5b74ecd874f982def0b3030 in term 1.
I20260807 22:57:49.651048 10967 raft_consensus.cc:2454] T ae32fd67806c4c82b19c11b9e5657b64 P 50d15ab368f242d6ba9acc047b4de0c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf1ab86fe5b74ecd874f982def0b3030 in term 1.
I20260807 22:57:49.651480 10640 leader_election.cc:302] T ae32fd67806c4c82b19c11b9e5657b64 P bf1ab86fe5b74ecd874f982def0b3030 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 28bb0f3e219e46489ba58f36a2f92f96, 50d15ab368f242d6ba9acc047b4de0c7, bf1ab86fe5b74ecd874f982def0b3030; no voters:
I20260807 22:57:49.651832 11090 raft_consensus.cc:2781] T ae32fd67806c4c82b19c11b9e5657b64 P bf1ab86fe5b74ecd874f982def0b3030 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:57:49.652333 11090 raft_consensus.cc:686] T ae32fd67806c4c82b19c11b9e5657b64 P bf1ab86fe5b74ecd874f982def0b3030 [term 1 LEADER]: Becoming Leader. State: Replica: bf1ab86fe5b74ecd874f982def0b3030, State: Running, Role: LEADER
I20260807 22:57:49.652813 11090 consensus_queue.cc:237] T ae32fd67806c4c82b19c11b9e5657b64 P bf1ab86fe5b74ecd874f982def0b3030 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28bb0f3e219e46489ba58f36a2f92f96" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 41971 } } peers { permanent_uuid: "bf1ab86fe5b74ecd874f982def0b3030" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 37595 } } peers { permanent_uuid: "50d15ab368f242d6ba9acc047b4de0c7" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 40037 } }
I20260807 22:57:49.658085 10242 catalog_manager.cc:5495] T ae32fd67806c4c82b19c11b9e5657b64 P bf1ab86fe5b74ecd874f982def0b3030 reported cstate change: term changed from 0 to 1, leader changed from <none> to bf1ab86fe5b74ecd874f982def0b3030 (127.25.94.2). New cstate: current_term: 1 leader_uuid: "bf1ab86fe5b74ecd874f982def0b3030" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28bb0f3e219e46489ba58f36a2f92f96" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 41971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf1ab86fe5b74ecd874f982def0b3030" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 37595 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "50d15ab368f242d6ba9acc047b4de0c7" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 40037 } health_report { overall_health: UNKNOWN } } }
W20260807 22:57:49.668535 11048 tablet.cc:2369] T ae32fd67806c4c82b19c11b9e5657b64 P 50d15ab368f242d6ba9acc047b4de0c7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 22:57:49.709007 11073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:49.704222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44824 (local address 127.25.94.2:37595)
0807 22:57:49.704417 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:49.704422 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:49.704438 (+ 16us) server_negotiation.cc:407] Connection header received
0807 22:57:49.704527 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:49.704531 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:49.704614 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:49.704736 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 22:57:49.705061 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:49.705819 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:49.707189 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:49.707582 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:49.707672 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:49.708081 (+ 409us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 22:57:49.708546 (+ 465us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 22:57:49.708649 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0807 22:57:49.708732 (+ 83us) server_negotiation.cc:299] Negotiation successful
0807 22:57:49.708791 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260807 22:57:49.717592 10517 raft_consensus.cc:1261] T ae32fd67806c4c82b19c11b9e5657b64 P 28bb0f3e219e46489ba58f36a2f92f96 [term 1 FOLLOWER]: Refusing update from remote peer bf1ab86fe5b74ecd874f982def0b3030: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 22:57:49.717589 10967 raft_consensus.cc:1261] T ae32fd67806c4c82b19c11b9e5657b64 P 50d15ab368f242d6ba9acc047b4de0c7 [term 1 FOLLOWER]: Refusing update from remote peer bf1ab86fe5b74ecd874f982def0b3030: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 22:57:49.718701 11090 consensus_queue.cc:1035] T ae32fd67806c4c82b19c11b9e5657b64 P bf1ab86fe5b74ecd874f982def0b3030 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50d15ab368f242d6ba9acc047b4de0c7" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 40037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 22:57:49.719053 11096 consensus_queue.cc:1035] T ae32fd67806c4c82b19c11b9e5657b64 P bf1ab86fe5b74ecd874f982def0b3030 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28bb0f3e219e46489ba58f36a2f92f96" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 41971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 22:57:49.748476 10241 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52710:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 22:57:49.748802 10241 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52710:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 22:57:49.752919 10241 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 667233c706e44aef884f4b51f73bdd6d: Sending DeleteTablet for 3 replicas of tablet ae32fd67806c4c82b19c11b9e5657b64
I20260807 22:57:49.754323 10709 tablet_service.cc:1514] Processing DeleteTablet for tablet ae32fd67806c4c82b19c11b9e5657b64 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:57:49 UTC) from {username='kudu', principal='kudu/127.25.94.62@KRBTEST.COM'} at 127.0.0.1:44808
I20260807 22:57:49.754350 10933 tablet_service.cc:1514] Processing DeleteTablet for tablet ae32fd67806c4c82b19c11b9e5657b64 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:57:49 UTC) from {username='kudu', principal='kudu/127.25.94.62@KRBTEST.COM'} at 127.0.0.1:43848
I20260807 22:57:49.755545 10478 tablet_service.cc:1514] Processing DeleteTablet for tablet ae32fd67806c4c82b19c11b9e5657b64 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:57:49 UTC) from {username='kudu', principal='kudu/127.25.94.62@KRBTEST.COM'} at 127.0.0.1:52718
I20260807 22:57:49.760700 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 10357
I20260807 22:57:49.764472 11122 tablet_replica.cc:330] T ae32fd67806c4c82b19c11b9e5657b64 P 28bb0f3e219e46489ba58f36a2f92f96: stopping tablet replica
I20260807 22:57:49.764809 11119 tablet_replica.cc:330] T ae32fd67806c4c82b19c11b9e5657b64 P bf1ab86fe5b74ecd874f982def0b3030: stopping tablet replica
I20260807 22:57:49.765093 11122 raft_consensus.cc:2229] T ae32fd67806c4c82b19c11b9e5657b64 P 28bb0f3e219e46489ba58f36a2f92f96 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 22:57:49.765466 11119 raft_consensus.cc:2229] T ae32fd67806c4c82b19c11b9e5657b64 P bf1ab86fe5b74ecd874f982def0b3030 [term 1 LEADER]: Raft consensus shutting down.
I20260807 22:57:49.765799 11122 raft_consensus.cc:2258] T ae32fd67806c4c82b19c11b9e5657b64 P 28bb0f3e219e46489ba58f36a2f92f96 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 22:57:49.766338 11119 raft_consensus.cc:2258] T ae32fd67806c4c82b19c11b9e5657b64 P bf1ab86fe5b74ecd874f982def0b3030 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 22:57:49.767140 11120 tablet_replica.cc:330] T ae32fd67806c4c82b19c11b9e5657b64 P 50d15ab368f242d6ba9acc047b4de0c7: stopping tablet replica
I20260807 22:57:49.767694 11120 raft_consensus.cc:2229] T ae32fd67806c4c82b19c11b9e5657b64 P 50d15ab368f242d6ba9acc047b4de0c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 22:57:49.768325 11120 raft_consensus.cc:2258] T ae32fd67806c4c82b19c11b9e5657b64 P 50d15ab368f242d6ba9acc047b4de0c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 22:57:49.769564 11119 ts_tablet_manager.cc:1905] T ae32fd67806c4c82b19c11b9e5657b64 P bf1ab86fe5b74ecd874f982def0b3030: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 22:57:49.770148 11122 ts_tablet_manager.cc:1905] T ae32fd67806c4c82b19c11b9e5657b64 P 28bb0f3e219e46489ba58f36a2f92f96: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 22:57:49.770848 11120 ts_tablet_manager.cc:1905] T ae32fd67806c4c82b19c11b9e5657b64 P 50d15ab368f242d6ba9acc047b4de0c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 22:57:49.777539 11122 ts_tablet_manager.cc:1918] T ae32fd67806c4c82b19c11b9e5657b64 P 28bb0f3e219e46489ba58f36a2f92f96: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 22:57:49.777813 11122 log.cc:1196] T ae32fd67806c4c82b19c11b9e5657b64 P 28bb0f3e219e46489ba58f36a2f92f96: Deleting WAL directory at /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786143361779964-25976-0/minicluster-data/ts-0/wal/wals/ae32fd67806c4c82b19c11b9e5657b64
I20260807 22:57:49.777997 11119 ts_tablet_manager.cc:1918] T ae32fd67806c4c82b19c11b9e5657b64 P bf1ab86fe5b74ecd874f982def0b3030: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 22:57:49.778199 11119 log.cc:1196] T ae32fd67806c4c82b19c11b9e5657b64 P bf1ab86fe5b74ecd874f982def0b3030: Deleting WAL directory at /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786143361779964-25976-0/minicluster-data/ts-1/wal/wals/ae32fd67806c4c82b19c11b9e5657b64
I20260807 22:57:49.778638 11122 ts_tablet_manager.cc:1939] T ae32fd67806c4c82b19c11b9e5657b64 P 28bb0f3e219e46489ba58f36a2f92f96: Deleting consensus metadata
I20260807 22:57:49.778781 11119 ts_tablet_manager.cc:1939] T ae32fd67806c4c82b19c11b9e5657b64 P bf1ab86fe5b74ecd874f982def0b3030: Deleting consensus metadata
I20260807 22:57:49.780805 10212 catalog_manager.cc:4841] TS 28bb0f3e219e46489ba58f36a2f92f96 (127.25.94.1:41971): tablet ae32fd67806c4c82b19c11b9e5657b64 (table test-table [id=a43b8e88d508469a9068be2caa4959b4]) successfully deleted
I20260807 22:57:49.780838 10213 catalog_manager.cc:4841] TS bf1ab86fe5b74ecd874f982def0b3030 (127.25.94.2:37595): tablet ae32fd67806c4c82b19c11b9e5657b64 (table test-table [id=a43b8e88d508469a9068be2caa4959b4]) successfully deleted
I20260807 22:57:49.782107 11120 ts_tablet_manager.cc:1918] T ae32fd67806c4c82b19c11b9e5657b64 P 50d15ab368f242d6ba9acc047b4de0c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 22:57:49.782392 11120 log.cc:1196] T ae32fd67806c4c82b19c11b9e5657b64 P 50d15ab368f242d6ba9acc047b4de0c7: Deleting WAL directory at /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786143361779964-25976-0/minicluster-data/ts-2/wal/wals/ae32fd67806c4c82b19c11b9e5657b64
I20260807 22:57:49.782960 11120 ts_tablet_manager.cc:1939] T ae32fd67806c4c82b19c11b9e5657b64 P 50d15ab368f242d6ba9acc047b4de0c7: Deleting consensus metadata
Aug 07 22:57:49 dist-test-slave-z8x9 krb5kdc[10160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143466, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.1@KRBTEST.COM
I20260807 22:57:49.785308 10214 catalog_manager.cc:4841] TS 50d15ab368f242d6ba9acc047b4de0c7 (127.25.94.3:40037): tablet ae32fd67806c4c82b19c11b9e5657b64 (table test-table [id=a43b8e88d508469a9068be2caa4959b4]) successfully deleted
I20260807 22:57:49.793352 11071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:49.762617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52720 (local address 127.25.94.1:41971)
0807 22:57:49.762832 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:49.762840 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:49.773276 (+ 10436us) server_negotiation.cc:407] Connection header received
0807 22:57:49.773758 (+ 482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:49.773764 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:49.773854 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:49.773944 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:49.775932 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:49.777109 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:49.779144 (+ 2035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:49.779815 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:49.779930 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:49.785250 (+ 5320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:57:49.785275 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:57:49.785296 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:57:49.785361 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:57:49.789806 (+ 4445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:49.790724 (+ 918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:49.790736 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:49.790739 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:49.790942 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:49.791363 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:49.791370 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:49.791373 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:49.791880 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:57:49.792254 (+ 374us) server_negotiation.cc:1035] Waiting for connection context
0807 22:57:49.792632 (+ 378us) server_negotiation.cc:299] Negotiation successful
0807 22:57:49.792897 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260807 22:57:49.794286 10597 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 22:57:49.895666 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 10357
I20260807 22:57:49.923259 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 10612
Aug 07 22:57:49 dist-test-slave-z8x9 krb5kdc[10160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143466, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.2@KRBTEST.COM
I20260807 22:57:49.941720 11073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:49.924081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44836 (local address 127.25.94.2:37595)
0807 22:57:49.924308 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:49.924314 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:49.924348 (+ 34us) server_negotiation.cc:407] Connection header received
0807 22:57:49.924419 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:49.924424 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:49.924505 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:49.924644 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:49.926213 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:49.927476 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:49.928905 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:49.929752 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:49.929910 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:49.934342 (+ 4432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:57:49.934364 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:57:49.934387 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:57:49.934448 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:57:49.938741 (+ 4293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:49.939474 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:49.939484 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:49.939488 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:49.939665 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:49.940033 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:49.940039 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:49.940042 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:49.940511 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:57:49.940876 (+ 365us) server_negotiation.cc:1035] Waiting for connection context
0807 22:57:49.941050 (+ 174us) server_negotiation.cc:299] Negotiation successful
0807 22:57:49.941314 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260807 22:57:49.942590 10814 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 22:57:50.060835 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 10612
I20260807 22:57:50.085759 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 10830
Aug 07 22:57:50 dist-test-slave-z8x9 krb5kdc[10160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143466, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.3@KRBTEST.COM
I20260807 22:57:50.102624 11079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:57:50.086567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43852 (local address 127.25.94.3:40037)
0807 22:57:50.086787 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0807 22:57:50.086794 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 22:57:50.086904 (+ 110us) server_negotiation.cc:407] Connection header received
0807 22:57:50.087122 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:57:50.087127 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:57:50.087228 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:57:50.087381 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:57:50.088702 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:50.089524 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:50.090804 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:57:50.091193 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:57:50.091346 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:57:50.096157 (+ 4811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:57:50.096174 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:57:50.096191 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:57:50.096228 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:57:50.099204 (+ 2976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:50.100161 (+ 957us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:50.100170 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:50.100172 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:50.100344 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:57:50.100776 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:57:50.100781 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:57:50.100787 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:57:50.101306 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:57:50.101649 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0807 22:57:50.102008 (+ 359us) server_negotiation.cc:299] Negotiation successful
0807 22:57:50.102242 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260807 22:57:50.103562 11038 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 22:57:50.197816 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 10830
I20260807 22:57:50.222286 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 10178
I20260807 22:57:50.223564 10294 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 22:57:50.336051 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 10178
2026-08-07T22:57:50Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (5799 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786143361779964-25976-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 22:57:50 dist-test-slave-z8x9 krb5kdc[11135](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 22:57:50 dist-test-slave-z8x9 krb5kdc[11135](info): set up 2 sockets
Aug 07 22:57:50 dist-test-slave-z8x9 krb5kdc[11135](info): commencing operation
krb5kdc: starting...
W20260807 22:57:51.716405 25976 mini_kdc.cc:121] Time spent starting KDC: real 1.325s user 0.001s sys 0.036s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786143361779964-25976-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786143361779964-25976-0/krb5kdc/test-user.keytab.
Aug 07 22:57:51 dist-test-slave-z8x9 krb5kdc[11135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143471, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T22:57:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T22:57:51Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.25.94.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.25.94.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.25.94.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786143361779964-25976-0/krb5kdc/rangeradmin_127.25.94.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.25.94.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786143361779964-25976-0/krb5kdc/rangeradmin_127.25.94.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.25.94.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.25.94.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.25.94.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786143361779964-25976-0/krb5kdc/rangerlookup_127.25.94.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.25.94.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786143361779964-25976-0/krb5kdc/rangerlookup_127.25.94.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.25.94.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.25.94.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.94.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786143361779964-25976-0/krb5kdc/HTTP_127.25.94.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.25.94.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786143361779964-25976-0/krb5kdc/HTTP_127.25.94.20@KRBTEST.COM.keytab.
I20260807 22:57:52.121013 25976 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786143361779964-25976-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-tasknHbun_/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786143361779964-25976-0/minicluster-data/postgres -l logfile start
2026-08-07 22:57:54.874 UTC [11188] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-07 22:57:54.874 UTC [11188] LOG: listening on IPv4 address "127.25.94.20", port 43919
2026-08-07 22:57:54.878 UTC [11188] LOG: listening on Unix socket "/tmp/.s.PGSQL.43919"
2026-08-07 22:57:54.902 UTC [11195] LOG: database system was shut down at 2026-08-07 22:57:53 UTC
2026-08-07 22:57:54.906 UTC [11188] LOG: database system is ready to accept connections
I20260807 22:57:56.133908 25976 mini_postgres.cc:96] Postgres bound to 43919
2026-08-07 22:57:56.159 UTC [11210] FATAL: database "slave" does not exist
127.25.94.20:43919 - accepting connections
I20260807 22:57:56.160339 25976 mini_ranger.cc:161] Starting Ranger...
I20260807 22:57:56.197968 25976 mini_ranger.cc:84] Created miniranger Postgres user
I20260807 22:57:56.759621 25976 mini_ranger.cc:87] Created ranger Postgres database
I20260807 22:57:56.759847 25976 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786143361779964-25976-0/minicluster-data/ranger-admin
2026-08-07 22:57:57,123 [I] DB FLAVOR :POSTGRES
2026-08-07 22:57:57,124 [I] --------- Verifying Ranger DB connection ---------
2026-08-07 22:57:57,124 [I] Checking connection..
2026-08-07 22:57:57,124 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknHbun_/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknHbun_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.94.20:43919/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-07 22:57:57,408 [I] Checking connection passed.
2026-08-07 22:57:57,409 [I] --------- Verifying version history table ---------
2026-08-07 22:57:57,409 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknHbun_/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknHbun_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.94.20:43919/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-07 22:57:57,673 [I] Table x_db_version_h does not exist in database ranger
2026-08-07 22:57:57,673 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknHbun_/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknHbun_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.94.20:43919/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-07 22:57:57,950 [I] Table x_db_version_h does not exist in database ranger
2026-08-07 22:57:57,950 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-07 22:57:57,950 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknHbun_/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknHbun_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.94.20:43919/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasknHbun_/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-07 22:57:58.220 UTC [11325] WARNING: there is no transaction in progress
2026-08-07 22:57:58,231 [I] create_dbversion_catalog.sql file imported successfully
2026-08-07 22:57:58,232 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknHbun_/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknHbun_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.94.20:43919/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-07 22:57:58,501 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-07 22:57:58,501 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-07 22:57:58,502 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknHbun_/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknHbun_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.94.20:43919/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-07 22:57:58,751 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknHbun_/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknHbun_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.94.20:43919/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-07 22:57:59,005 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknHbun_/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknHbun_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.94.20:43919/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-z8x9.c.gcp-upstream.internal','N') ;"
2026-08-07 22:57:59,257 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknHbun_/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknHbun_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.94.20:43919/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-07 22:57:59,517 [I] Table x_portal_user does not exist in database ranger
2026-08-07 22:57:59,517 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknHbun_/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknHbun_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.94.20:43919/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-07 22:57:59,791 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-07 22:57:59,792 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknHbun_/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknHbun_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.94.20:43919/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-07 22:58:00,050 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknHbun_/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknHbun_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.94.20:43919/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-07 22:58:00,311 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-07 22:58:00,311 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknHbun_/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknHbun_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.94.20:43919/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasknHbun_/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-07 22:58:01.374 UTC [11525] WARNING: there is no transaction in progress
2026-08-07 22:58:01.392 UTC [11525] WARNING: there is no transaction in progress
2026-08-07 22:58:01.409 UTC [11525] WARNING: there is no transaction in progress
2026-08-07 22:58:01.425 UTC [11525] WARNING: there is no transaction in progress
2026-08-07 22:58:01.443 UTC [11525] WARNING: there is no transaction in progress
2026-08-07 22:58:01.461 UTC [11525] WARNING: there is no transaction in progress
2026-08-07 22:58:01.517 UTC [11525] WARNING: there is no transaction in progress
2026-08-07 22:58:01.525 UTC [11525] WARNING: there is no transaction in progress
2026-08-07 22:58:01.540 UTC [11525] WARNING: there is no transaction in progress
2026-08-07 22:58:01.552 UTC [11525] WARNING: there is no transaction in progress
2026-08-07 22:58:01.563 UTC [11525] WARNING: there is no transaction in progress
2026-08-07 22:58:01.577 UTC [11525] WARNING: there is no transaction in progress
2026-08-07 22:58:01.588 UTC [11525] WARNING: there is no transaction in progress
2026-08-07 22:58:01.595 UTC [11525] WARNING: there is no transaction in progress
2026-08-07 22:58:01.602 UTC [11525] WARNING: there is no transaction in progress
2026-08-07 22:58:01.609 UTC [11525] WARNING: there is no transaction in progress
2026-08-07 22:58:02,354 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-07 22:58:02,354 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknHbun_/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknHbun_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.94.20:43919/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-07 22:58:02,621 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-07 22:58:02,621 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknHbun_/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknHbun_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.94.20:43919/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-07 22:58:02,881 [I] Table x_portal_user already exists in database 'ranger'
2026-08-07 22:58:02,881 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknHbun_/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknHbun_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.94.20:43919/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-07 22:58:03,147 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-07 22:58:03,147 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknHbun_/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknHbun_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.94.20:43919/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-07 22:58:03,435 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknHbun_/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknHbun_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.94.20:43919/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-07 22:58:03,705 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknHbun_/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknHbun_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.94.20:43919/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-z8x9.c.gcp-upstream.internal';"
2026-08-07 22:58:03,973 [I] CORE_DB_SCHEMA import status has been updated
2026-08-07 22:58:03,974 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknHbun_/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknHbun_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.94.20:43919/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-07 22:58:04,254 [I] DB_PATCHES have already been applied
I20260807 22:58:04.261277 25976 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786143361779964-25976-0/minicluster-data/ranger-admin:/tmp/dist-test-tasknHbun_/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-tasknHbun_/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-tasknHbun_/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknHbun_/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260807 22:58:04.261376 25976 mini_ranger.cc:193] Using host: 127.25.94.20
I20260807 22:58:04.282572 25976 mini_ranger.cc:227] Ranger admin URL: http://127.25.94.20:37373
Aug 07, 2026 10:58:04 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 07, 2026 10:58:04 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 07, 2026 10:58:04 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786143361779964-25976-0/minicluster-data/ranger-admin/ews/webapp
Aug 07, 2026 10:58:04 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786143361779964-25976-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 07, 2026 10:58:04 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786143361779964-25976-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 07, 2026 10:58:05 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786143361779964-25976-0/minicluster-data/ranger-admin/ews/webapp].
Aug 07, 2026 10:58:05 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260807 22:58:05.751483 25976 mini_ranger.cc:160] Time spent starting Ranger: real 9.591s user 0.003s sys 0.059s
Aug 07, 2026 10:58:26 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 07, 2026 10:58:26 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.MetricsREST
Aug 07, 2026 10:58:26 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 07, 2026 10:58:26 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 07, 2026 10:58:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 07, 2026 10:58:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 07, 2026 10:58:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 07, 2026 10:58:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 07, 2026 10:58:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 07, 2026 10:58:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 07, 2026 10:58:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 07, 2026 10:58:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 07, 2026 10:58:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 07, 2026 10:58:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 07, 2026 10:58:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 07, 2026 10:58:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 07, 2026 10:58:26 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 07, 2026 10:58:27 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260807 22:58:28.335438 25976 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.25.94.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.25.94.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.25.94.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786143361779964-25976-0/krb5kdc/rangerkms_127.25.94.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.25.94.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786143361779964-25976-0/krb5kdc/rangerkms_127.25.94.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.25.94.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.25.94.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.94.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786143361779964-25976-0/krb5kdc/HTTP_127.25.94.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.25.94.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786143361779964-25976-0/krb5kdc/HTTP_127.25.94.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 07 22:58:28 dist-test-slave-z8x9 krb5kdc[11135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143508, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260807 22:58:28.563233 25976 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260807 22:58:28.597260 25976 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260807 22:58:29.576858 25976 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260807 22:58:29.577054 25976 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786143361779964-25976-0/minicluster-data/ranger-kms
I20260807 22:58:29.577241 25976 mini_ranger_kms.cc:227] Using postgres at 127.25.94.20:43919
2026-08-07 22:58:30,114 [I] DB FLAVOR :POSTGRES
2026-08-07 22:58:30,115 [I] --------- Verifying Ranger DB connection ---------
2026-08-07 22:58:30,115 [I] Checking connection
2026-08-07 22:58:30,115 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknHbun_/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786143361779964-25976-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.94.20:43919/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-07 22:58:30,389 [I] connection success
2026-08-07 22:58:30,389 [I] --------- Verifying Ranger DB tables ---------
2026-08-07 22:58:30,389 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-07 22:58:30,389 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknHbun_/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786143361779964-25976-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.94.20:43919/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-07 22:58:30,637 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-07 22:58:30,638 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-07 22:58:30,638 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-07 22:58:30,638 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknHbun_/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786143361779964-25976-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.94.20:43919/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786143361779964-25976-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-07 22:58:30,917 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260807 22:58:31.466552 25976 mini_ranger_kms.cc:310] Created kms service
I20260807 22:58:31.678735 25976 mini_ranger_kms.cc:326] Created kudu user
I20260807 22:58:31.726580 25976 mini_ranger_kms.cc:343] Created rangerkms user
Aug 07 22:58:31 dist-test-slave-z8x9 krb5kdc[11135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143508, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.25.94.20@KRBTEST.COM
I20260807 22:58:32.175197 25976 mini_ranger_kms.cc:384] Added ranger policy
I20260807 22:58:32.176716 25976 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786143361779964-25976-0/minicluster-data/ranger-kms:/tmp/dist-test-tasknHbun_/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknHbun_/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-tasknHbun_/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260807 22:58:32.176834 25976 mini_ranger_kms.cc:240] Using host: 127.25.94.20
I20260807 22:58:32.200821 25976 mini_ranger_kms.cc:276] Ranger KMS PID: 11830
I20260807 22:58:32.201069 25976 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.25.94.20:53091
Aug 07, 2026 10:58:33 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 07, 2026 10:58:33 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 07, 2026 10:58:33 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786143361779964-25976-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 07, 2026 10:58:33 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786143361779964-25976-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 07, 2026 10:58:33 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786143361779964-25976-0/minicluster-data/ranger-kms/ews/webapp].
Aug 07, 2026 10:58:33 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 07 22:58:34 dist-test-slave-z8x9 krb5kdc[11135](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786143514, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.25.94.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Aug 07 22:58:34 dist-test-slave-z8x9 krb5kdc[11135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143514, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.25.94.20@KRBTEST.COM for HTTP/127.25.94.20@KRBTEST.COM
I20260807 22:58:35.081866 25976 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.519s user 0.145s sys 0.421s
I20260807 22:58:35.082235 25976 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260807 22:58:35.082444 25976 mini_ranger_kms.cc:401] 127.25.94.20:53091/kms/v1/keys
Aug 07 22:58:35 dist-test-slave-z8x9 krb5kdc[11135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143514, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.25.94.20@KRBTEST.COM for HTTP/127.25.94.20@KRBTEST.COM
Aug 07 22:58:37 dist-test-slave-z8x9 krb5kdc[11135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143517, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.25.94.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.62.keytab.
Entry for principal kudu/127.25.94.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.62.keytab.
I20260807 22:58:37.828369 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786143361779964-25976-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786143361779964-25976-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786143361779964-25976-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.62:40491
--webserver_interface=127.25.94.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.62.keytab
--principal=kudu/127.25.94.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.20:33895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.94.62:40491
--ranger_config_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786143361779964-25976-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.25.94.20:53091/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786143361779964-25976-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786143361779964-25976-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786143361779964-25976-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:58:38.333089 11923 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:58:38.333508 11923 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:58:38.333668 11923 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:58:38.341905 11923 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 22:58:38.342108 11923 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:58:38.342192 11923 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:58:38.342268 11923 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 22:58:38.342326 11923 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 22:58:38.353905 11923 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:33895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786143361779964-25976-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786143361779964-25976-0/minicluster-data/master-0/wal
--ranger_kms_url=127.25.94.20:53091/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.25.94.62:40491
--ranger_config_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786143361779964-25976-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.62.keytab
--principal=kudu/127.25.94.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.62:40491
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.94.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786143361779964-25976-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 22:58:38.355506 11923 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:58:38.357028 11923 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:58:38.371353 11932 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:58:38.371402 11929 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:58:38.371371 11930 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:58:38.372066 11923 server_base.cc:1034] running on GCE node
I20260807 22:58:38.372997 11923 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:58:38.374782 11923 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:58:38.376087 11923 hybrid_clock.cc:648] HybridClock initialized: now 1786143518376008 us; error 56 us; skew 500 ppm
I20260807 22:58:38.376533 11923 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 22:58:38 dist-test-slave-z8x9 krb5kdc[11135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143518, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:58:38.380404 11923 init.cc:377] Logged in from keytab as kudu/127.25.94.62@KRBTEST.COM (short username kudu)
I20260807 22:58:38.382273 11923 webserver.cc:460] Webserver started at http://127.25.94.62:40469/ using document root <none> and password file <none>
I20260807 22:58:38.382978 11923 fs_manager.cc:359] Metadata directory not provided
I20260807 22:58:38.383124 11923 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:58:38.383441 11923 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 07 22:58:38 dist-test-slave-z8x9 krb5kdc[11135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143518, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.62@KRBTEST.COM for HTTP/127.25.94.20@KRBTEST.COM
I20260807 22:58:38.453271 11923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786143361779964-25976-0/minicluster-data/master-0/data/instance:
uuid: "aa0561b0c48b4fcf8c7b4dc414279f91"
format_stamp: "Formatted at 2026-08-07 22:58:38 on dist-test-slave-z8x9"
server_key: "167f2711a2acc15a3061d6474aea5b54"
server_key_iv: "1421a70633ff25275ed119313c74b000"
server_key_version: "kuduclusterkey@0"
I20260807 22:58:38.454491 11923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786143361779964-25976-0/minicluster-data/master-0/wal/instance:
uuid: "aa0561b0c48b4fcf8c7b4dc414279f91"
format_stamp: "Formatted at 2026-08-07 22:58:38 on dist-test-slave-z8x9"
server_key: "167f2711a2acc15a3061d6474aea5b54"
server_key_iv: "1421a70633ff25275ed119313c74b000"
server_key_version: "kuduclusterkey@0"
I20260807 22:58:38.462949 11923 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.006s sys 0.002s
I20260807 22:58:38.482177 11941 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:58:38.483917 11923 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260807 22:58:38.484153 11923 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786143361779964-25976-0/minicluster-data/master-0/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786143361779964-25976-0/minicluster-data/master-0/wal
uuid: "aa0561b0c48b4fcf8c7b4dc414279f91"
format_stamp: "Formatted at 2026-08-07 22:58:38 on dist-test-slave-z8x9"
server_key: "167f2711a2acc15a3061d6474aea5b54"
server_key_iv: "1421a70633ff25275ed119313c74b000"
server_key_version: "kuduclusterkey@0"
I20260807 22:58:38.484417 11923 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786143361779964-25976-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786143361779964-25976-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786143361779964-25976-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:58:38.497849 11923 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:58:38.503209 11923 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:58:38.503711 11923 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:58:38.540154 11923 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.62:40491
I20260807 22:58:38.540324 12032 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.62:40491 every 8 connection(s)
I20260807 22:58:38.542032 11923 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
I20260807 22:58:38.546653 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 11923
I20260807 22:58:38.546927 25976 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786143361779964-25976-0/minicluster-data/master-0/wal/instance
I20260807 22:58:38.548084 25976 external_mini_cluster.cc:1428] Setting key 3c550d3b8886eb701a4bfc6d60c0717e
I20260807 22:58:38.548290 12035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 07 22:58:38 dist-test-slave-z8x9 krb5kdc[11135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143517, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.62@KRBTEST.COM
I20260807 22:58:38.566723 12035 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa0561b0c48b4fcf8c7b4dc414279f91: Bootstrap starting.
I20260807 22:58:38.571940 12035 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aa0561b0c48b4fcf8c7b4dc414279f91: Neither blocks nor log segments found. Creating new log.
I20260807 22:58:38.574190 12035 log.cc:824] T 00000000000000000000000000000000 P aa0561b0c48b4fcf8c7b4dc414279f91: Log is configured to *not* fsync() on all Append() calls
I20260807 22:58:38.578094 12043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:58:38.550748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46006 (local address 127.25.94.62:40491)
0807 22:58:38.552300 (+ 1552us) server_negotiation.cc:206] Beginning negotiation
0807 22:58:38.552315 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0807 22:58:38.552367 (+ 52us) server_negotiation.cc:407] Connection header received
0807 22:58:38.555501 (+ 3134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:58:38.555527 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:58:38.555983 (+ 456us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:58:38.556435 (+ 452us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:58:38.557977 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:58:38.559803 (+ 1826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:58:38.561374 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:58:38.562346 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:58:38.562598 (+ 252us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:58:38.567897 (+ 5299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:58:38.567907 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:58:38.567923 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:58:38.567995 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:58:38.572582 (+ 4587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:58:38.573658 (+ 1076us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:58:38.573666 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:58:38.573672 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:58:38.573873 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:58:38.574372 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:58:38.574381 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:58:38.574383 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:58:38.575528 (+ 1145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:58:38.575989 (+ 461us) server_negotiation.cc:1035] Waiting for connection context
0807 22:58:38.576414 (+ 425us) server_negotiation.cc:299] Negotiation successful
0807 22:58:38.576897 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1262,"thread_start_us":928,"threads_started":1}
I20260807 22:58:38.578904 12035 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa0561b0c48b4fcf8c7b4dc414279f91: No bootstrap required, opened a new log
I20260807 22:58:38.589514 12035 raft_consensus.cc:348] T 00000000000000000000000000000000 P aa0561b0c48b4fcf8c7b4dc414279f91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa0561b0c48b4fcf8c7b4dc414279f91" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 40491 } }
I20260807 22:58:38.589941 12035 raft_consensus.cc:374] T 00000000000000000000000000000000 P aa0561b0c48b4fcf8c7b4dc414279f91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:58:38.590116 12035 raft_consensus.cc:729] T 00000000000000000000000000000000 P aa0561b0c48b4fcf8c7b4dc414279f91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa0561b0c48b4fcf8c7b4dc414279f91, State: Initialized, Role: FOLLOWER
I20260807 22:58:38.590857 12035 consensus_queue.cc:260] T 00000000000000000000000000000000 P aa0561b0c48b4fcf8c7b4dc414279f91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa0561b0c48b4fcf8c7b4dc414279f91" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 40491 } }
I20260807 22:58:38.591269 12035 raft_consensus.cc:388] T 00000000000000000000000000000000 P aa0561b0c48b4fcf8c7b4dc414279f91 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 22:58:38.591470 12035 raft_consensus.cc:482] T 00000000000000000000000000000000 P aa0561b0c48b4fcf8c7b4dc414279f91 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 22:58:38.591699 12035 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aa0561b0c48b4fcf8c7b4dc414279f91 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:58:38.594947 12035 raft_consensus.cc:504] T 00000000000000000000000000000000 P aa0561b0c48b4fcf8c7b4dc414279f91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa0561b0c48b4fcf8c7b4dc414279f91" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 40491 } }
I20260807 22:58:38.595532 12035 leader_election.cc:302] T 00000000000000000000000000000000 P aa0561b0c48b4fcf8c7b4dc414279f91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aa0561b0c48b4fcf8c7b4dc414279f91; no voters:
I20260807 22:58:38.596760 12035 leader_election.cc:288] T 00000000000000000000000000000000 P aa0561b0c48b4fcf8c7b4dc414279f91 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 22:58:38.597026 12045 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aa0561b0c48b4fcf8c7b4dc414279f91 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:58:38.599261 12045 raft_consensus.cc:686] T 00000000000000000000000000000000 P aa0561b0c48b4fcf8c7b4dc414279f91 [term 1 LEADER]: Becoming Leader. State: Replica: aa0561b0c48b4fcf8c7b4dc414279f91, State: Running, Role: LEADER
I20260807 22:58:38.600065 12045 consensus_queue.cc:237] T 00000000000000000000000000000000 P aa0561b0c48b4fcf8c7b4dc414279f91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa0561b0c48b4fcf8c7b4dc414279f91" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 40491 } }
I20260807 22:58:38.600862 12035 sys_catalog.cc:564] T 00000000000000000000000000000000 P aa0561b0c48b4fcf8c7b4dc414279f91 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 22:58:38.605334 12046 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa0561b0c48b4fcf8c7b4dc414279f91 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aa0561b0c48b4fcf8c7b4dc414279f91" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa0561b0c48b4fcf8c7b4dc414279f91" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 40491 } } }
I20260807 22:58:38.604794 12047 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa0561b0c48b4fcf8c7b4dc414279f91 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aa0561b0c48b4fcf8c7b4dc414279f91. Latest consensus state: current_term: 1 leader_uuid: "aa0561b0c48b4fcf8c7b4dc414279f91" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa0561b0c48b4fcf8c7b4dc414279f91" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 40491 } } }
I20260807 22:58:38.606709 12047 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa0561b0c48b4fcf8c7b4dc414279f91 [sys.catalog]: This master's current role is: LEADER
I20260807 22:58:38.606818 12046 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa0561b0c48b4fcf8c7b4dc414279f91 [sys.catalog]: This master's current role is: LEADER
I20260807 22:58:38.610316 12054 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 22:58:38.618400 12054 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 22:58:38.630353 12035 ranger_client.cc:318] Using new properties file: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786143361779964-25976-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260807 22:58:38.635360 12054 catalog_manager.cc:1347] Generated new cluster ID: 0ad03d4850664deb9ecaaf5b3cc26e43
I20260807 22:58:38.635601 12054 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 22:58:38.673467 12054 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 22:58:38.677006 12054 catalog_manager.cc:1504] Loading token signing keys...
I20260807 22:58:38.689486 12054 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aa0561b0c48b4fcf8c7b4dc414279f91: Generated new TSK 0
I20260807 22:58:38.691028 12054 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.94.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.1.keytab.
Entry for principal kudu/127.25.94.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.1.keytab.
I20260807 22:58:39.586115 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786143361779964-25976-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786143361779964-25976-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786143361779964-25976-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786143361779964-25976-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.25.94.20:53091/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.1:0
--local_ip_for_outbound_sockets=127.25.94.1
--webserver_interface=127.25.94.1
--webserver_port=0
--tserver_master_addrs=127.25.94.62:40491
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.1.keytab
--principal=kudu/127.25.94.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.20:33895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786143361779964-25976-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786143361779964-25976-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786143361779964-25976-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 07 22:58:39 dist-test-slave-z8x9 krb5kdc[11135](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786143519, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260807 22:58:40.163612 12116 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:58:40.164124 12116 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:58:40.164343 12116 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:58:40.177598 12116 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:58:40.178114 12116 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:58:40.178545 12116 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.1
I20260807 22:58:40.198868 12116 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:33895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786143361779964-25976-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786143361779964-25976-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.25.94.20:53091/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.1.keytab
--principal=kudu/127.25.94.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786143361779964-25976-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.94.1
--webserver_port=0
--tserver_master_addrs=127.25.94.62:40491
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.1
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786143361779964-25976-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 22:58:40.200443 12116 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:58:40.202257 12116 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:58:40.228365 12140 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:58:40.228358 12143 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:58:40.231911 12137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:58:40.232491 12116 server_base.cc:1034] running on GCE node
I20260807 22:58:40.233443 12116 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:58:40.234676 12116 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:58:40.236176 12116 hybrid_clock.cc:648] HybridClock initialized: now 1786143520235969 us; error 55 us; skew 500 ppm
I20260807 22:58:40.237043 12116 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 22:58:40 dist-test-slave-z8x9 krb5kdc[11135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143520, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:58:40.242199 12116 init.cc:377] Logged in from keytab as kudu/127.25.94.1@KRBTEST.COM (short username kudu)
I20260807 22:58:40.244709 12116 webserver.cc:460] Webserver started at http://127.25.94.1:40713/ using document root <none> and password file <none>
I20260807 22:58:40.245591 12116 fs_manager.cc:359] Metadata directory not provided
I20260807 22:58:40.245724 12116 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:58:40.246168 12116 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 07 22:58:40 dist-test-slave-z8x9 krb5kdc[11135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143520, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.1@KRBTEST.COM for HTTP/127.25.94.20@KRBTEST.COM
I20260807 22:58:40.272576 12116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786143361779964-25976-0/minicluster-data/ts-0/data/instance:
uuid: "d84a367949804c4fa7cc45104cf515d1"
format_stamp: "Formatted at 2026-08-07 22:58:40 on dist-test-slave-z8x9"
server_key: "e8d47bff318d6afd56eeddc1a18f7728"
server_key_iv: "f7a4c14d668b5389ad496dcec10b6977"
server_key_version: "kuduclusterkey@0"
I20260807 22:58:40.274190 12116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786143361779964-25976-0/minicluster-data/ts-0/wal/instance:
uuid: "d84a367949804c4fa7cc45104cf515d1"
format_stamp: "Formatted at 2026-08-07 22:58:40 on dist-test-slave-z8x9"
server_key: "e8d47bff318d6afd56eeddc1a18f7728"
server_key_iv: "f7a4c14d668b5389ad496dcec10b6977"
server_key_version: "kuduclusterkey@0"
I20260807 22:58:40.284405 12116 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.004s sys 0.004s
I20260807 22:58:40.327448 12161 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:58:40.329517 12116 fs_manager.cc:725] Time spent opening block manager: real 0.012s user 0.006s sys 0.002s
I20260807 22:58:40.329938 12116 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786143361779964-25976-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786143361779964-25976-0/minicluster-data/ts-0/wal
uuid: "d84a367949804c4fa7cc45104cf515d1"
format_stamp: "Formatted at 2026-08-07 22:58:40 on dist-test-slave-z8x9"
server_key: "e8d47bff318d6afd56eeddc1a18f7728"
server_key_iv: "f7a4c14d668b5389ad496dcec10b6977"
server_key_version: "kuduclusterkey@0"
I20260807 22:58:40.330410 12116 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786143361779964-25976-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786143361779964-25976-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786143361779964-25976-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:58:40.361146 12116 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:58:40.367609 12116 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:58:40.368206 12116 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:58:40.372965 12116 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:58:40.378810 12116 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:58:40.379051 12116 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260807 22:58:40.379376 12116 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:58:40.379565 12116 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 07 22:58:40 dist-test-slave-z8x9 krb5kdc[11135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143519, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.62@KRBTEST.COM for HTTP/127.25.94.20@KRBTEST.COM
I20260807 22:58:40.760951 12343 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.1:46545 every 8 connection(s)
I20260807 22:58:40.761842 12116 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.1:46545
I20260807 22:58:40.764226 12116 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786143361779964-25976-0/minicluster-data/ts-0/data/info.pb
I20260807 22:58:40.770073 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 12116
I20260807 22:58:40.770377 25976 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786143361779964-25976-0/minicluster-data/ts-0/wal/instance
I20260807 22:58:40.771329 25976 external_mini_cluster.cc:1428] Setting key c2fe51d51ba740d77cc4f7eb8ba55d02
Aug 07 22:58:40 dist-test-slave-z8x9 krb5kdc[11135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143520, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.1@KRBTEST.COM for kudu/127.25.94.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.25.94.2@KRBTEST.COM; defaulting to no policy
I20260807 22:58:40.843859 12350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:58:40.774578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.1:38779 (local address 127.25.94.62:40491)
0807 22:58:40.807939 (+ 33361us) server_negotiation.cc:206] Beginning negotiation
0807 22:58:40.807946 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 22:58:40.809301 (+ 1355us) server_negotiation.cc:407] Connection header received
0807 22:58:40.819033 (+ 9732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:58:40.819041 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:58:40.819143 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:58:40.819228 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:58:40.822410 (+ 3182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:58:40.823875 (+ 1465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:58:40.826414 (+ 2539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:58:40.827395 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:58:40.827501 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:58:40.833971 (+ 6470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:58:40.833979 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:58:40.833984 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:58:40.834059 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:58:40.838743 (+ 4684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:58:40.839994 (+ 1251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:58:40.840001 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:58:40.840004 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:58:40.840266 (+ 262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:58:40.841064 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:58:40.841070 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:58:40.841074 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:58:40.841676 (+ 602us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:58:40.841850 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0807 22:58:40.843015 (+ 1165us) server_negotiation.cc:299] Negotiation successful
0807 22:58:40.843279 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33159,"thread_start_us":33342,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.2@KRBTEST.COM" created.
I20260807 22:58:40.845165 12348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:58:40.772782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:40491 (local address 127.25.94.1:38779)
0807 22:58:40.809003 (+ 36221us) negotiation.cc:107] Waiting for socket to connect
0807 22:58:40.809065 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0807 22:58:40.818603 (+ 9538us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:58:40.819908 (+ 1305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:58:40.819921 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:58:40.820799 (+ 878us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:58:40.822159 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:58:40.822190 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:58:40.824171 (+ 1981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:58:40.824176 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:58:40.826222 (+ 2046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:58:40.826239 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:58:40.828031 (+ 1792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:58:40.828038 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:58:40.828391 (+ 353us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:58:40.829560 (+ 1169us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:58:40.829592 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:58:40.833623 (+ 4031us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:58:40.839011 (+ 5388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:58:40.839020 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:58:40.839040 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:58:40.839798 (+ 758us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:58:40.840437 (+ 639us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:58:40.840442 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:58:40.840446 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:58:40.840892 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:58:40.841848 (+ 956us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:58:40.841856 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:58:40.842361 (+ 505us) client_negotiation.cc:769] Sending connection context
0807 22:58:40.842741 (+ 380us) client_negotiation.cc:240] Negotiation successful
0807 22:58:40.843409 (+ 668us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35811,"thread_start_us":42355,"threads_started":1}
I20260807 22:58:40.847831 12347 heartbeater.cc:347] Connected to a master server at 127.25.94.62:40491
I20260807 22:58:40.848240 12347 heartbeater.cc:464] Registering TS with master...
I20260807 22:58:40.849642 12347 heartbeater.cc:511] Master 127.25.94.62:40491 requested a full tablet report, sending...
I20260807 22:58:40.852687 11966 ts_manager.cc:194] Registered new tserver with Master: d84a367949804c4fa7cc45104cf515d1 (127.25.94.1:46545)
I20260807 22:58:40.855337 11966 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.94.1@KRBTEST.COM'} at 127.25.94.1:38779
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.2.keytab.
Entry for principal kudu/127.25.94.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.2.keytab.
W20260807 22:58:40.884433 25976 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.25.94.2: real 0.109s user 0.002s sys 0.025s
I20260807 22:58:40.885414 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786143361779964-25976-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786143361779964-25976-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786143361779964-25976-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786143361779964-25976-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.25.94.20:53091/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.2:0
--local_ip_for_outbound_sockets=127.25.94.2
--webserver_interface=127.25.94.2
--webserver_port=0
--tserver_master_addrs=127.25.94.62:40491
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.2.keytab
--principal=kudu/127.25.94.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.20:33895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786143361779964-25976-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786143361779964-25976-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786143361779964-25976-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 07 22:58:40 dist-test-slave-z8x9 krb5kdc[11135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143519, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.62@KRBTEST.COM for HTTP/127.25.94.20@KRBTEST.COM
W20260807 22:58:41.760078 12362 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:58:41.760530 12362 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:58:41.760825 12362 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:58:41.772292 12362 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:58:41.772584 12362 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:58:41.772851 12362 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.2
I20260807 22:58:41.787858 12362 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:33895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786143361779964-25976-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786143361779964-25976-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.25.94.20:53091/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.2.keytab
--principal=kudu/127.25.94.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786143361779964-25976-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.94.2
--webserver_port=0
--tserver_master_addrs=127.25.94.62:40491
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.2
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786143361779964-25976-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 22:58:41.789579 12362 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:58:41.791247 12362 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:58:41.814245 12393 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:58:41.815943 12398 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:58:41.815970 12394 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:58:41.817307 12362 server_base.cc:1034] running on GCE node
I20260807 22:58:41.818470 12362 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:58:41.820173 12362 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:58:41.821550 12362 hybrid_clock.cc:648] HybridClock initialized: now 1786143521821317 us; error 135 us; skew 500 ppm
I20260807 22:58:41.822158 12362 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 22:58:41 dist-test-slave-z8x9 krb5kdc[11135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143521, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:58:41.829095 12362 init.cc:377] Logged in from keytab as kudu/127.25.94.2@KRBTEST.COM (short username kudu)
I20260807 22:58:41.831975 12362 webserver.cc:460] Webserver started at http://127.25.94.2:34519/ using document root <none> and password file <none>
I20260807 22:58:41.833261 12362 fs_manager.cc:359] Metadata directory not provided
I20260807 22:58:41.833528 12362 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:58:41.833995 12362 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 07 22:58:41 dist-test-slave-z8x9 krb5kdc[11135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143521, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.2@KRBTEST.COM for HTTP/127.25.94.20@KRBTEST.COM
I20260807 22:58:41.856004 12362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786143361779964-25976-0/minicluster-data/ts-1/data/instance:
uuid: "4b1638ba908c459d8eeb299a892bf252"
format_stamp: "Formatted at 2026-08-07 22:58:41 on dist-test-slave-z8x9"
server_key: "6bfff496c73338bb0ef4d9c0df31a380"
server_key_iv: "41688952a290b63c830c7b32a22e0017"
server_key_version: "kuduclusterkey@0"
I20260807 22:58:41.857614 12362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786143361779964-25976-0/minicluster-data/ts-1/wal/instance:
uuid: "4b1638ba908c459d8eeb299a892bf252"
format_stamp: "Formatted at 2026-08-07 22:58:41 on dist-test-slave-z8x9"
server_key: "6bfff496c73338bb0ef4d9c0df31a380"
server_key_iv: "41688952a290b63c830c7b32a22e0017"
server_key_version: "kuduclusterkey@0"
I20260807 22:58:41.859673 12347 heartbeater.cc:503] Master 127.25.94.62:40491 was elected leader, sending a full tablet report...
I20260807 22:58:41.868582 12362 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.008s sys 0.000s
I20260807 22:58:41.891443 12408 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:58:41.893674 12362 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.003s sys 0.000s
I20260807 22:58:41.894101 12362 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786143361779964-25976-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786143361779964-25976-0/minicluster-data/ts-1/wal
uuid: "4b1638ba908c459d8eeb299a892bf252"
format_stamp: "Formatted at 2026-08-07 22:58:41 on dist-test-slave-z8x9"
server_key: "6bfff496c73338bb0ef4d9c0df31a380"
server_key_iv: "41688952a290b63c830c7b32a22e0017"
server_key_version: "kuduclusterkey@0"
I20260807 22:58:41.894515 12362 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786143361779964-25976-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786143361779964-25976-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786143361779964-25976-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:58:41.916523 12362 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:58:41.923882 12362 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:58:41.924573 12362 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:58:41.927261 12362 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:58:41.931054 12362 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:58:41.931260 12362 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:58:41.931572 12362 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:58:41.931734 12362 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:58:42.153518 12362 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.2:37509
I20260807 22:58:42.153712 12626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.2:37509 every 8 connection(s)
I20260807 22:58:42.155366 12362 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786143361779964-25976-0/minicluster-data/ts-1/data/info.pb
I20260807 22:58:42.163777 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 12362
I20260807 22:58:42.164026 25976 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786143361779964-25976-0/minicluster-data/ts-1/wal/instance
I20260807 22:58:42.164860 25976 external_mini_cluster.cc:1428] Setting key 41d5debced19129124def3eaf51b89aa
Aug 07 22:58:42 dist-test-slave-z8x9 krb5kdc[11135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143521, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.2@KRBTEST.COM for kudu/127.25.94.62@KRBTEST.COM
I20260807 22:58:42.210898 12640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:58:42.166686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.2:45635 (local address 127.25.94.62:40491)
0807 22:58:42.178029 (+ 11343us) server_negotiation.cc:206] Beginning negotiation
0807 22:58:42.178037 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 22:58:42.178060 (+ 23us) server_negotiation.cc:407] Connection header received
0807 22:58:42.178124 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:58:42.178129 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:58:42.178235 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:58:42.178450 (+ 215us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:58:42.182425 (+ 3975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:58:42.183958 (+ 1533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:58:42.187268 (+ 3310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:58:42.188097 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:58:42.188242 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:58:42.199979 (+ 11737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:58:42.199990 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:58:42.199998 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:58:42.200055 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:58:42.205498 (+ 5443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:58:42.206918 (+ 1420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:58:42.206928 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:58:42.206934 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:58:42.207190 (+ 256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:58:42.207948 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:58:42.207956 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:58:42.207963 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:58:42.208464 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:58:42.208641 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0807 22:58:42.209946 (+ 1305us) server_negotiation.cc:299] Negotiation successful
0807 22:58:42.210250 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11103,"thread_start_us":10872,"threads_started":1}
I20260807 22:58:42.214306 12637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:58:42.162233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:40491 (local address 127.25.94.2:45635)
0807 22:58:42.170200 (+ 7967us) negotiation.cc:107] Waiting for socket to connect
0807 22:58:42.170271 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0807 22:58:42.174153 (+ 3882us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:58:42.179279 (+ 5126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:58:42.179298 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:58:42.180640 (+ 1342us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:58:42.182119 (+ 1479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:58:42.182156 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:58:42.184224 (+ 2068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:58:42.184232 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:58:42.187003 (+ 2771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:58:42.187054 (+ 51us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:58:42.188818 (+ 1764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:58:42.188831 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:58:42.189501 (+ 670us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:58:42.195578 (+ 6077us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:58:42.195622 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:58:42.199605 (+ 3983us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:58:42.205786 (+ 6181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:58:42.205800 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:58:42.205824 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:58:42.206728 (+ 904us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:58:42.207359 (+ 631us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:58:42.207366 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:58:42.207373 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:58:42.207793 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:58:42.208773 (+ 980us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:58:42.208785 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:58:42.209372 (+ 587us) client_negotiation.cc:769] Sending connection context
0807 22:58:42.211615 (+ 2243us) client_negotiation.cc:240] Negotiation successful
0807 22:58:42.212386 (+ 771us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7534,"thread_start_us":7094,"threads_started":1}
I20260807 22:58:42.216775 12629 heartbeater.cc:347] Connected to a master server at 127.25.94.62:40491
I20260807 22:58:42.217341 12629 heartbeater.cc:464] Registering TS with master...
I20260807 22:58:42.218686 12629 heartbeater.cc:511] Master 127.25.94.62:40491 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.25.94.3@KRBTEST.COM; defaulting to no policy
I20260807 22:58:42.220805 11966 ts_manager.cc:194] Registered new tserver with Master: 4b1638ba908c459d8eeb299a892bf252 (127.25.94.2:37509)
I20260807 22:58:42.222301 11966 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.94.2@KRBTEST.COM'} at 127.25.94.2:45635
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.3.keytab.
Entry for principal kudu/127.25.94.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.3.keytab.
I20260807 22:58:42.261951 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786143361779964-25976-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786143361779964-25976-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786143361779964-25976-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786143361779964-25976-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.25.94.20:53091/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.3:0
--local_ip_for_outbound_sockets=127.25.94.3
--webserver_interface=127.25.94.3
--webserver_port=0
--tserver_master_addrs=127.25.94.62:40491
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.3.keytab
--principal=kudu/127.25.94.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.20:33895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786143361779964-25976-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786143361779964-25976-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786143361779964-25976-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:58:42.862561 12643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:58:42.863025 12643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:58:42.863288 12643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:58:42.873340 12643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:58:42.873538 12643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:58:42.873768 12643 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.3
I20260807 22:58:42.886044 12643 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:33895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786143361779964-25976-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786143361779964-25976-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.25.94.20:53091/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.3.keytab
--principal=kudu/127.25.94.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786143361779964-25976-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.94.3
--webserver_port=0
--tserver_master_addrs=127.25.94.62:40491
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.3
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786143361779964-25976-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 22:58:42.887539 12643 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:58:42.889147 12643 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:58:42.905977 12657 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:58:42.906165 12656 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:58:42.906205 12660 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:58:42.908861 12643 server_base.cc:1034] running on GCE node
I20260807 22:58:42.909679 12643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:58:42.910735 12643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:58:42.911991 12643 hybrid_clock.cc:648] HybridClock initialized: now 1786143522911933 us; error 59 us; skew 500 ppm
I20260807 22:58:42.912381 12643 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 22:58:42 dist-test-slave-z8x9 krb5kdc[11135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143522, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:58:42.916718 12643 init.cc:377] Logged in from keytab as kudu/127.25.94.3@KRBTEST.COM (short username kudu)
I20260807 22:58:42.918877 12643 webserver.cc:460] Webserver started at http://127.25.94.3:38985/ using document root <none> and password file <none>
I20260807 22:58:42.919893 12643 fs_manager.cc:359] Metadata directory not provided
I20260807 22:58:42.920065 12643 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:58:42.920392 12643 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 07 22:58:42 dist-test-slave-z8x9 krb5kdc[11135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143522, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.3@KRBTEST.COM for HTTP/127.25.94.20@KRBTEST.COM
I20260807 22:58:42.935696 12643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786143361779964-25976-0/minicluster-data/ts-2/data/instance:
uuid: "ad8e3419f4a245e39ce61bcf0be910bf"
format_stamp: "Formatted at 2026-08-07 22:58:42 on dist-test-slave-z8x9"
server_key: "d941540e8aacdf970edbd705c58708aa"
server_key_iv: "4ef82df105c6d33dc8664e7aca91dcc6"
server_key_version: "kuduclusterkey@0"
I20260807 22:58:42.936650 12643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786143361779964-25976-0/minicluster-data/ts-2/wal/instance:
uuid: "ad8e3419f4a245e39ce61bcf0be910bf"
format_stamp: "Formatted at 2026-08-07 22:58:42 on dist-test-slave-z8x9"
server_key: "d941540e8aacdf970edbd705c58708aa"
server_key_iv: "4ef82df105c6d33dc8664e7aca91dcc6"
server_key_version: "kuduclusterkey@0"
I20260807 22:58:42.942914 12643 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.002s
I20260807 22:58:42.960150 12668 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:58:42.962162 12643 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260807 22:58:42.962457 12643 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786143361779964-25976-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786143361779964-25976-0/minicluster-data/ts-2/wal
uuid: "ad8e3419f4a245e39ce61bcf0be910bf"
format_stamp: "Formatted at 2026-08-07 22:58:42 on dist-test-slave-z8x9"
server_key: "d941540e8aacdf970edbd705c58708aa"
server_key_iv: "4ef82df105c6d33dc8664e7aca91dcc6"
server_key_version: "kuduclusterkey@0"
I20260807 22:58:42.962771 12643 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786143361779964-25976-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786143361779964-25976-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786143361779964-25976-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:58:43.001642 12643 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:58:43.007016 12643 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:58:43.007458 12643 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:58:43.008841 12643 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:58:43.010876 12643 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:58:43.010982 12643 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:58:43.011111 12643 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:58:43.011166 12643 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:58:43.081692 12643 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.3:40071
I20260807 22:58:43.081740 12872 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.3:40071 every 8 connection(s)
I20260807 22:58:43.083328 12643 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786143361779964-25976-0/minicluster-data/ts-2/data/info.pb
I20260807 22:58:43.090296 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 12643
I20260807 22:58:43.090497 25976 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786143361779964-25976-0/minicluster-data/ts-2/wal/instance
I20260807 22:58:43.091012 25976 external_mini_cluster.cc:1428] Setting key f36b7e24a086f5bd24f1fd2fefad2280
Aug 07 22:58:43 dist-test-slave-z8x9 krb5kdc[11135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143522, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.3@KRBTEST.COM for kudu/127.25.94.62@KRBTEST.COM
I20260807 22:58:43.113011 12876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:58:43.087420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.3:43915 (local address 127.25.94.62:40491)
0807 22:58:43.088474 (+ 1054us) server_negotiation.cc:206] Beginning negotiation
0807 22:58:43.088482 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 22:58:43.089627 (+ 1145us) server_negotiation.cc:407] Connection header received
0807 22:58:43.092228 (+ 2601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:58:43.092235 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:58:43.092355 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:58:43.092949 (+ 594us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:58:43.095010 (+ 2061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:58:43.096130 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:58:43.098443 (+ 2313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:58:43.099110 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:58:43.099271 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:58:43.104416 (+ 5145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:58:43.104424 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:58:43.104427 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:58:43.104472 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:58:43.109157 (+ 4685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:58:43.110296 (+ 1139us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:58:43.110301 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:58:43.110303 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:58:43.110512 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:58:43.111062 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:58:43.111065 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:58:43.111067 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:58:43.111403 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:58:43.111537 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0807 22:58:43.112383 (+ 846us) server_negotiation.cc:299] Negotiation successful
0807 22:58:43.112626 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":828,"thread_start_us":621,"threads_started":1}
I20260807 22:58:43.113900 12877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:58:43.087935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:40491 (local address 127.25.94.3:43915)
0807 22:58:43.089418 (+ 1483us) negotiation.cc:107] Waiting for socket to connect
0807 22:58:43.089464 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0807 22:58:43.091974 (+ 2510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:58:43.092848 (+ 874us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:58:43.092861 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:58:43.093661 (+ 800us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:58:43.094810 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:58:43.094840 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:58:43.096346 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:58:43.096352 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:58:43.098196 (+ 1844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:58:43.098234 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:58:43.099468 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:58:43.099478 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:58:43.099858 (+ 380us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:58:43.100788 (+ 930us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:58:43.100813 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:58:43.104098 (+ 3285us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:58:43.109405 (+ 5307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:58:43.109415 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:58:43.109432 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:58:43.110142 (+ 710us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:58:43.110616 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:58:43.110620 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:58:43.110622 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:58:43.110934 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:58:43.111528 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:58:43.111535 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:58:43.111892 (+ 357us) client_negotiation.cc:769] Sending connection context
0807 22:58:43.112152 (+ 260us) client_negotiation.cc:240] Negotiation successful
0807 22:58:43.112585 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1180,"thread_start_us":846,"threads_started":1}
I20260807 22:58:43.115700 12874 heartbeater.cc:347] Connected to a master server at 127.25.94.62:40491
I20260807 22:58:43.116072 12874 heartbeater.cc:464] Registering TS with master...
I20260807 22:58:43.117033 12874 heartbeater.cc:511] Master 127.25.94.62:40491 requested a full tablet report, sending...
I20260807 22:58:43.119185 11966 ts_manager.cc:194] Registered new tserver with Master: ad8e3419f4a245e39ce61bcf0be910bf (127.25.94.3:40071)
I20260807 22:58:43.120388 11966 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.94.3@KRBTEST.COM'} at 127.25.94.3:43915
I20260807 22:58:43.121618 25976 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 22:58:43.145531 12876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:58:43.128323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42582 (local address 127.25.94.62:40491)
0807 22:58:43.128532 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0807 22:58:43.128540 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 22:58:43.129150 (+ 610us) server_negotiation.cc:407] Connection header received
0807 22:58:43.129415 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:58:43.129419 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:58:43.129542 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:58:43.129663 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:58:43.131704 (+ 2041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:58:43.132694 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:58:43.134506 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:58:43.135012 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:58:43.135160 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:58:43.138452 (+ 3292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:58:43.138457 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:58:43.138461 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:58:43.138500 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:58:43.142030 (+ 3530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:58:43.142938 (+ 908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:58:43.142944 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:58:43.142946 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:58:43.143092 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:58:43.143584 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:58:43.143590 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:58:43.143593 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:58:43.143911 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:58:43.144288 (+ 377us) server_negotiation.cc:1035] Waiting for connection context
0807 22:58:43.144788 (+ 500us) server_negotiation.cc:299] Negotiation successful
0807 22:58:43.145103 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260807 22:58:43.150854 11966 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42582:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 22:58:43.160591 11966 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 22:58:43.207800 12895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:58:43.182212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50224 (local address 127.25.94.2:37509)
0807 22:58:43.187761 (+ 5549us) server_negotiation.cc:206] Beginning negotiation
0807 22:58:43.187771 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 22:58:43.187806 (+ 35us) server_negotiation.cc:407] Connection header received
0807 22:58:43.187934 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:58:43.187943 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:58:43.188272 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:58:43.189076 (+ 804us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:58:43.193316 (+ 4240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:58:43.198951 (+ 5635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:58:43.204140 (+ 5189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:58:43.205666 (+ 1526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:58:43.205843 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:58:43.206193 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0807 22:58:43.206972 (+ 779us) server_negotiation.cc:299] Negotiation successful
0807 22:58:43.207189 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5345,"thread_start_us":5178,"threads_started":1}
I20260807 22:58:43.208026 12898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:58:43.183394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38868 (local address 127.25.94.3:40071)
0807 22:58:43.191551 (+ 8157us) server_negotiation.cc:206] Beginning negotiation
0807 22:58:43.191566 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0807 22:58:43.191621 (+ 55us) server_negotiation.cc:407] Connection header received
0807 22:58:43.191764 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:58:43.191777 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:58:43.192239 (+ 462us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:58:43.192568 (+ 329us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:58:43.194336 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:58:43.197497 (+ 3161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:58:43.203016 (+ 5519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:58:43.205602 (+ 2586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:58:43.205755 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:58:43.206106 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0807 22:58:43.207492 (+ 1386us) server_negotiation.cc:299] Negotiation successful
0807 22:58:43.207655 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7848,"thread_start_us":7569,"threads_started":1}
I20260807 22:58:43.209730 12893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:58:43.181977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.2:37509 (local address 127.0.0.1:50224)
0807 22:58:43.186481 (+ 4504us) negotiation.cc:107] Waiting for socket to connect
0807 22:58:43.186531 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0807 22:58:43.186766 (+ 235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:58:43.188577 (+ 1811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:58:43.188586 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:58:43.188620 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:58:43.190250 (+ 1630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:58:43.190267 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:58:43.199298 (+ 9031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:58:43.199307 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:58:43.203919 (+ 4612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:58:43.203932 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:58:43.206410 (+ 2478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:58:43.206418 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:58:43.206639 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:58:43.206724 (+ 85us) client_negotiation.cc:769] Sending connection context
0807 22:58:43.206886 (+ 162us) client_negotiation.cc:240] Negotiation successful
0807 22:58:43.207571 (+ 685us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4122,"mutex_wait_us":524,"thread_start_us":3878,"threads_started":1}
I20260807 22:58:43.209746 12897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:58:43.182467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.3:40071 (local address 127.0.0.1:38868)
0807 22:58:43.187866 (+ 5399us) negotiation.cc:107] Waiting for socket to connect
0807 22:58:43.187891 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0807 22:58:43.188069 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:58:43.192598 (+ 4529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:58:43.192607 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:58:43.192653 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:58:43.193855 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:58:43.193878 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:58:43.197783 (+ 3905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:58:43.197794 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:58:43.202727 (+ 4933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:58:43.202762 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:58:43.206469 (+ 3707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:58:43.206480 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:58:43.206696 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:58:43.206733 (+ 37us) client_negotiation.cc:769] Sending connection context
0807 22:58:43.206851 (+ 118us) client_negotiation.cc:240] Negotiation successful
0807 22:58:43.207152 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5148,"mutex_wait_us":32,"thread_start_us":4907,"threads_started":1}
I20260807 22:58:43.214007 12901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:58:43.188785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.1:46545 (local address 127.0.0.1:49032)
0807 22:58:43.200542 (+ 11757us) negotiation.cc:107] Waiting for socket to connect
0807 22:58:43.200569 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0807 22:58:43.200829 (+ 260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:58:43.202073 (+ 1244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:58:43.202078 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:58:43.202111 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:58:43.203412 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:58:43.203444 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:58:43.207017 (+ 3573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:58:43.207023 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:58:43.210673 (+ 3650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:58:43.210691 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:58:43.212832 (+ 2141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:58:43.212842 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:58:43.213089 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:58:43.213124 (+ 35us) client_negotiation.cc:769] Sending connection context
0807 22:58:43.213260 (+ 136us) client_negotiation.cc:240] Negotiation successful
0807 22:58:43.213410 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11592,"mutex_wait_us":23,"thread_start_us":11407,"threads_started":1}
I20260807 22:58:43.214038 12900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:58:43.189161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49032 (local address 127.25.94.1:46545)
0807 22:58:43.201059 (+ 11898us) server_negotiation.cc:206] Beginning negotiation
0807 22:58:43.201071 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 22:58:43.201108 (+ 37us) server_negotiation.cc:407] Connection header received
0807 22:58:43.201290 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:58:43.201300 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:58:43.201748 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:58:43.202135 (+ 387us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:58:43.203745 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:58:43.206200 (+ 2455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:58:43.210924 (+ 4724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:58:43.212398 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:58:43.212567 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:58:43.212903 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0807 22:58:43.213326 (+ 423us) server_negotiation.cc:299] Negotiation successful
0807 22:58:43.213605 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11538,"thread_start_us":11794,"threads_started":1}
I20260807 22:58:43.215090 12502 tablet_service.cc:1467] Processing CreateTablet for tablet 1b9460e145db47f983002febe388784e (DEFAULT_TABLE table=test-table [id=9355baf3dd764951a4bf64a6ed24e358]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:58:43.215740 12748 tablet_service.cc:1467] Processing CreateTablet for tablet 1b9460e145db47f983002febe388784e (DEFAULT_TABLE table=test-table [id=9355baf3dd764951a4bf64a6ed24e358]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:58:43.217417 12502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b9460e145db47f983002febe388784e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:58:43.217873 12748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b9460e145db47f983002febe388784e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:58:43.219125 12238 tablet_service.cc:1467] Processing CreateTablet for tablet 1b9460e145db47f983002febe388784e (DEFAULT_TABLE table=test-table [id=9355baf3dd764951a4bf64a6ed24e358]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:58:43.221088 12238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b9460e145db47f983002febe388784e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:58:43.232472 12629 heartbeater.cc:503] Master 127.25.94.62:40491 was elected leader, sending a full tablet report...
I20260807 22:58:43.236733 12902 tablet_bootstrap.cc:492] T 1b9460e145db47f983002febe388784e P 4b1638ba908c459d8eeb299a892bf252: Bootstrap starting.
I20260807 22:58:43.239632 12905 tablet_bootstrap.cc:492] T 1b9460e145db47f983002febe388784e P ad8e3419f4a245e39ce61bcf0be910bf: Bootstrap starting.
I20260807 22:58:43.243512 12906 tablet_bootstrap.cc:492] T 1b9460e145db47f983002febe388784e P d84a367949804c4fa7cc45104cf515d1: Bootstrap starting.
I20260807 22:58:43.244997 12902 tablet_bootstrap.cc:654] T 1b9460e145db47f983002febe388784e P 4b1638ba908c459d8eeb299a892bf252: Neither blocks nor log segments found. Creating new log.
I20260807 22:58:43.246809 12905 tablet_bootstrap.cc:654] T 1b9460e145db47f983002febe388784e P ad8e3419f4a245e39ce61bcf0be910bf: Neither blocks nor log segments found. Creating new log.
I20260807 22:58:43.247560 12902 log.cc:824] T 1b9460e145db47f983002febe388784e P 4b1638ba908c459d8eeb299a892bf252: Log is configured to *not* fsync() on all Append() calls
I20260807 22:58:43.248121 12906 tablet_bootstrap.cc:654] T 1b9460e145db47f983002febe388784e P d84a367949804c4fa7cc45104cf515d1: Neither blocks nor log segments found. Creating new log.
I20260807 22:58:43.249343 12905 log.cc:824] T 1b9460e145db47f983002febe388784e P ad8e3419f4a245e39ce61bcf0be910bf: Log is configured to *not* fsync() on all Append() calls
I20260807 22:58:43.250710 12906 log.cc:824] T 1b9460e145db47f983002febe388784e P d84a367949804c4fa7cc45104cf515d1: Log is configured to *not* fsync() on all Append() calls
I20260807 22:58:43.252462 12902 tablet_bootstrap.cc:492] T 1b9460e145db47f983002febe388784e P 4b1638ba908c459d8eeb299a892bf252: No bootstrap required, opened a new log
I20260807 22:58:43.252835 12902 ts_tablet_manager.cc:1397] T 1b9460e145db47f983002febe388784e P 4b1638ba908c459d8eeb299a892bf252: Time spent bootstrapping tablet: real 0.016s user 0.013s sys 0.000s
I20260807 22:58:43.254659 12906 tablet_bootstrap.cc:492] T 1b9460e145db47f983002febe388784e P d84a367949804c4fa7cc45104cf515d1: No bootstrap required, opened a new log
I20260807 22:58:43.255016 12906 ts_tablet_manager.cc:1397] T 1b9460e145db47f983002febe388784e P d84a367949804c4fa7cc45104cf515d1: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20260807 22:58:43.255635 12905 tablet_bootstrap.cc:492] T 1b9460e145db47f983002febe388784e P ad8e3419f4a245e39ce61bcf0be910bf: No bootstrap required, opened a new log
I20260807 22:58:43.256108 12905 ts_tablet_manager.cc:1397] T 1b9460e145db47f983002febe388784e P ad8e3419f4a245e39ce61bcf0be910bf: Time spent bootstrapping tablet: real 0.017s user 0.011s sys 0.000s
I20260807 22:58:43.263981 12902 raft_consensus.cc:348] T 1b9460e145db47f983002febe388784e P 4b1638ba908c459d8eeb299a892bf252 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b1638ba908c459d8eeb299a892bf252" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 37509 } } peers { permanent_uuid: "ad8e3419f4a245e39ce61bcf0be910bf" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 40071 } } peers { permanent_uuid: "d84a367949804c4fa7cc45104cf515d1" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 46545 } }
I20260807 22:58:43.264596 12902 raft_consensus.cc:374] T 1b9460e145db47f983002febe388784e P 4b1638ba908c459d8eeb299a892bf252 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:58:43.264822 12902 raft_consensus.cc:729] T 1b9460e145db47f983002febe388784e P 4b1638ba908c459d8eeb299a892bf252 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b1638ba908c459d8eeb299a892bf252, State: Initialized, Role: FOLLOWER
I20260807 22:58:43.265766 12902 consensus_queue.cc:260] T 1b9460e145db47f983002febe388784e P 4b1638ba908c459d8eeb299a892bf252 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b1638ba908c459d8eeb299a892bf252" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 37509 } } peers { permanent_uuid: "ad8e3419f4a245e39ce61bcf0be910bf" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 40071 } } peers { permanent_uuid: "d84a367949804c4fa7cc45104cf515d1" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 46545 } }
I20260807 22:58:43.268356 12902 ts_tablet_manager.cc:1428] T 1b9460e145db47f983002febe388784e P 4b1638ba908c459d8eeb299a892bf252: Time spent starting tablet: real 0.015s user 0.005s sys 0.009s
I20260807 22:58:43.271698 12906 raft_consensus.cc:348] T 1b9460e145db47f983002febe388784e P d84a367949804c4fa7cc45104cf515d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b1638ba908c459d8eeb299a892bf252" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 37509 } } peers { permanent_uuid: "ad8e3419f4a245e39ce61bcf0be910bf" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 40071 } } peers { permanent_uuid: "d84a367949804c4fa7cc45104cf515d1" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 46545 } }
I20260807 22:58:43.272459 12906 raft_consensus.cc:374] T 1b9460e145db47f983002febe388784e P d84a367949804c4fa7cc45104cf515d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:58:43.272725 12906 raft_consensus.cc:729] T 1b9460e145db47f983002febe388784e P d84a367949804c4fa7cc45104cf515d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d84a367949804c4fa7cc45104cf515d1, State: Initialized, Role: FOLLOWER
I20260807 22:58:43.272838 12905 raft_consensus.cc:348] T 1b9460e145db47f983002febe388784e P ad8e3419f4a245e39ce61bcf0be910bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b1638ba908c459d8eeb299a892bf252" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 37509 } } peers { permanent_uuid: "ad8e3419f4a245e39ce61bcf0be910bf" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 40071 } } peers { permanent_uuid: "d84a367949804c4fa7cc45104cf515d1" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 46545 } }
I20260807 22:58:43.273496 12905 raft_consensus.cc:374] T 1b9460e145db47f983002febe388784e P ad8e3419f4a245e39ce61bcf0be910bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:58:43.273653 12905 raft_consensus.cc:729] T 1b9460e145db47f983002febe388784e P ad8e3419f4a245e39ce61bcf0be910bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad8e3419f4a245e39ce61bcf0be910bf, State: Initialized, Role: FOLLOWER
I20260807 22:58:43.274443 12905 consensus_queue.cc:260] T 1b9460e145db47f983002febe388784e P ad8e3419f4a245e39ce61bcf0be910bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b1638ba908c459d8eeb299a892bf252" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 37509 } } peers { permanent_uuid: "ad8e3419f4a245e39ce61bcf0be910bf" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 40071 } } peers { permanent_uuid: "d84a367949804c4fa7cc45104cf515d1" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 46545 } }
I20260807 22:58:43.275188 12906 consensus_queue.cc:260] T 1b9460e145db47f983002febe388784e P d84a367949804c4fa7cc45104cf515d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b1638ba908c459d8eeb299a892bf252" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 37509 } } peers { permanent_uuid: "ad8e3419f4a245e39ce61bcf0be910bf" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 40071 } } peers { permanent_uuid: "d84a367949804c4fa7cc45104cf515d1" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 46545 } }
I20260807 22:58:43.276131 12874 heartbeater.cc:503] Master 127.25.94.62:40491 was elected leader, sending a full tablet report...
I20260807 22:58:43.277634 12905 ts_tablet_manager.cc:1428] T 1b9460e145db47f983002febe388784e P ad8e3419f4a245e39ce61bcf0be910bf: Time spent starting tablet: real 0.021s user 0.012s sys 0.008s
I20260807 22:58:43.278196 12906 ts_tablet_manager.cc:1428] T 1b9460e145db47f983002febe388784e P d84a367949804c4fa7cc45104cf515d1: Time spent starting tablet: real 0.023s user 0.022s sys 0.001s
W20260807 22:58:43.307370 12353 tablet.cc:2369] T 1b9460e145db47f983002febe388784e P d84a367949804c4fa7cc45104cf515d1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 22:58:43.336542 12875 tablet.cc:2369] T 1b9460e145db47f983002febe388784e P ad8e3419f4a245e39ce61bcf0be910bf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 22:58:43.397392 12915 raft_consensus.cc:482] T 1b9460e145db47f983002febe388784e P ad8e3419f4a245e39ce61bcf0be910bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 22:58:43.397867 12915 raft_consensus.cc:504] T 1b9460e145db47f983002febe388784e P ad8e3419f4a245e39ce61bcf0be910bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b1638ba908c459d8eeb299a892bf252" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 37509 } } peers { permanent_uuid: "ad8e3419f4a245e39ce61bcf0be910bf" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 40071 } } peers { permanent_uuid: "d84a367949804c4fa7cc45104cf515d1" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 46545 } }
I20260807 22:58:43.400166 12915 leader_election.cc:288] T 1b9460e145db47f983002febe388784e P ad8e3419f4a245e39ce61bcf0be910bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4b1638ba908c459d8eeb299a892bf252 (127.25.94.2:37509), d84a367949804c4fa7cc45104cf515d1 (127.25.94.1:46545)
I20260807 22:58:43.408614 12877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:58:43.400389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.2:37509 (local address 127.25.94.3:47581)
0807 22:58:43.400587 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0807 22:58:43.400605 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 22:58:43.400816 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:58:43.401849 (+ 1033us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:58:43.401863 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:58:43.401889 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:58:43.402170 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:58:43.402180 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:58:43.404141 (+ 1961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:58:43.404148 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:58:43.406355 (+ 2207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:58:43.406368 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:58:43.407778 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:58:43.407784 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:58:43.408006 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:58:43.408032 (+ 26us) client_negotiation.cc:769] Sending connection context
0807 22:58:43.408142 (+ 110us) client_negotiation.cc:240] Negotiation successful
0807 22:58:43.408286 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260807 22:58:43.408629 12895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:58:43.400927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.3:47581 (local address 127.25.94.2:37509)
0807 22:58:43.401246 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0807 22:58:43.401254 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 22:58:43.401275 (+ 21us) server_negotiation.cc:407] Connection header received
0807 22:58:43.401351 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:58:43.401358 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:58:43.401435 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:58:43.401549 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:58:43.402375 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:58:43.403940 (+ 1565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:58:43.406513 (+ 2573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:58:43.407581 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:58:43.407767 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:58:43.407890 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0807 22:58:43.408170 (+ 280us) server_negotiation.cc:299] Negotiation successful
0807 22:58:43.408298 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260807 22:58:43.410223 12900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:58:43.401392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.3:52371 (local address 127.25.94.1:46545)
0807 22:58:43.401612 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0807 22:58:43.401618 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 22:58:43.402251 (+ 633us) server_negotiation.cc:407] Connection header received
0807 22:58:43.402516 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:58:43.402521 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:58:43.402596 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:58:43.402666 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:58:43.403925 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:58:43.405344 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:58:43.407958 (+ 2614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:58:43.409155 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:58:43.409381 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:58:43.409515 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0807 22:58:43.409824 (+ 309us) server_negotiation.cc:299] Negotiation successful
0807 22:58:43.409952 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260807 22:58:43.410365 12920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:58:43.400820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.1:46545 (local address 127.25.94.3:52371)
0807 22:58:43.402147 (+ 1327us) negotiation.cc:107] Waiting for socket to connect
0807 22:58:43.402163 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 22:58:43.402322 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:58:43.402725 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:58:43.402730 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:58:43.402756 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:58:43.403735 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:58:43.403750 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:58:43.405597 (+ 1847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:58:43.405601 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:58:43.407760 (+ 2159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:58:43.407774 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:58:43.409478 (+ 1704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:58:43.409486 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:58:43.409659 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:58:43.409684 (+ 25us) client_negotiation.cc:769] Sending connection context
0807 22:58:43.409795 (+ 111us) client_negotiation.cc:240] Negotiation successful
0807 22:58:43.409937 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1111,"thread_start_us":821,"threads_started":1}
I20260807 22:58:43.410020 12538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b9460e145db47f983002febe388784e" candidate_uuid: "ad8e3419f4a245e39ce61bcf0be910bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b1638ba908c459d8eeb299a892bf252" is_pre_election: true
I20260807 22:58:43.410944 12538 raft_consensus.cc:2454] T 1b9460e145db47f983002febe388784e P 4b1638ba908c459d8eeb299a892bf252 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad8e3419f4a245e39ce61bcf0be910bf in term 0.
I20260807 22:58:43.412128 12676 leader_election.cc:302] T 1b9460e145db47f983002febe388784e P ad8e3419f4a245e39ce61bcf0be910bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b1638ba908c459d8eeb299a892bf252, ad8e3419f4a245e39ce61bcf0be910bf; no voters:
I20260807 22:58:43.411959 12272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b9460e145db47f983002febe388784e" candidate_uuid: "ad8e3419f4a245e39ce61bcf0be910bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d84a367949804c4fa7cc45104cf515d1" is_pre_election: true
I20260807 22:58:43.412657 12915 raft_consensus.cc:2781] T 1b9460e145db47f983002febe388784e P ad8e3419f4a245e39ce61bcf0be910bf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 22:58:43.412882 12915 raft_consensus.cc:482] T 1b9460e145db47f983002febe388784e P ad8e3419f4a245e39ce61bcf0be910bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 22:58:43.413036 12915 raft_consensus.cc:3037] T 1b9460e145db47f983002febe388784e P ad8e3419f4a245e39ce61bcf0be910bf [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:58:43.412868 12272 raft_consensus.cc:2454] T 1b9460e145db47f983002febe388784e P d84a367949804c4fa7cc45104cf515d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad8e3419f4a245e39ce61bcf0be910bf in term 0.
I20260807 22:58:43.416301 12915 raft_consensus.cc:504] T 1b9460e145db47f983002febe388784e P ad8e3419f4a245e39ce61bcf0be910bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b1638ba908c459d8eeb299a892bf252" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 37509 } } peers { permanent_uuid: "ad8e3419f4a245e39ce61bcf0be910bf" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 40071 } } peers { permanent_uuid: "d84a367949804c4fa7cc45104cf515d1" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 46545 } }
I20260807 22:58:43.417411 12915 leader_election.cc:288] T 1b9460e145db47f983002febe388784e P ad8e3419f4a245e39ce61bcf0be910bf [CANDIDATE]: Term 1 election: Requested vote from peers 4b1638ba908c459d8eeb299a892bf252 (127.25.94.2:37509), d84a367949804c4fa7cc45104cf515d1 (127.25.94.1:46545)
I20260807 22:58:43.417970 12538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b9460e145db47f983002febe388784e" candidate_uuid: "ad8e3419f4a245e39ce61bcf0be910bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b1638ba908c459d8eeb299a892bf252"
I20260807 22:58:43.418069 12272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b9460e145db47f983002febe388784e" candidate_uuid: "ad8e3419f4a245e39ce61bcf0be910bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d84a367949804c4fa7cc45104cf515d1"
I20260807 22:58:43.418243 12538 raft_consensus.cc:3037] T 1b9460e145db47f983002febe388784e P 4b1638ba908c459d8eeb299a892bf252 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:58:43.418334 12272 raft_consensus.cc:3037] T 1b9460e145db47f983002febe388784e P d84a367949804c4fa7cc45104cf515d1 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:58:43.421402 12538 raft_consensus.cc:2454] T 1b9460e145db47f983002febe388784e P 4b1638ba908c459d8eeb299a892bf252 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad8e3419f4a245e39ce61bcf0be910bf in term 1.
I20260807 22:58:43.421553 12272 raft_consensus.cc:2454] T 1b9460e145db47f983002febe388784e P d84a367949804c4fa7cc45104cf515d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad8e3419f4a245e39ce61bcf0be910bf in term 1.
I20260807 22:58:43.422067 12676 leader_election.cc:302] T 1b9460e145db47f983002febe388784e P ad8e3419f4a245e39ce61bcf0be910bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b1638ba908c459d8eeb299a892bf252, ad8e3419f4a245e39ce61bcf0be910bf; no voters:
I20260807 22:58:43.422472 12915 raft_consensus.cc:2781] T 1b9460e145db47f983002febe388784e P ad8e3419f4a245e39ce61bcf0be910bf [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:58:43.423156 12915 raft_consensus.cc:686] T 1b9460e145db47f983002febe388784e P ad8e3419f4a245e39ce61bcf0be910bf [term 1 LEADER]: Becoming Leader. State: Replica: ad8e3419f4a245e39ce61bcf0be910bf, State: Running, Role: LEADER
I20260807 22:58:43.423817 12915 consensus_queue.cc:237] T 1b9460e145db47f983002febe388784e P ad8e3419f4a245e39ce61bcf0be910bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b1638ba908c459d8eeb299a892bf252" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 37509 } } peers { permanent_uuid: "ad8e3419f4a245e39ce61bcf0be910bf" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 40071 } } peers { permanent_uuid: "d84a367949804c4fa7cc45104cf515d1" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 46545 } }
W20260807 22:58:43.426966 12632 tablet.cc:2369] T 1b9460e145db47f983002febe388784e P 4b1638ba908c459d8eeb299a892bf252: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 22:58:43.429956 11965 catalog_manager.cc:5495] T 1b9460e145db47f983002febe388784e P ad8e3419f4a245e39ce61bcf0be910bf reported cstate change: term changed from 0 to 1, leader changed from <none> to ad8e3419f4a245e39ce61bcf0be910bf (127.25.94.3). New cstate: current_term: 1 leader_uuid: "ad8e3419f4a245e39ce61bcf0be910bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b1638ba908c459d8eeb299a892bf252" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 37509 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad8e3419f4a245e39ce61bcf0be910bf" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 40071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d84a367949804c4fa7cc45104cf515d1" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 46545 } health_report { overall_health: UNKNOWN } } }
I20260807 22:58:43.505606 12898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:58:43.498906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38884 (local address 127.25.94.3:40071)
0807 22:58:43.499095 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0807 22:58:43.499099 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 22:58:43.499118 (+ 19us) server_negotiation.cc:407] Connection header received
0807 22:58:43.499301 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:58:43.499305 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:58:43.499397 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:58:43.499528 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 22:58:43.499972 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:58:43.501025 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:58:43.503213 (+ 2188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:58:43.503713 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:58:43.503827 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:58:43.504349 (+ 522us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 22:58:43.504919 (+ 570us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 22:58:43.505029 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0807 22:58:43.505139 (+ 110us) server_negotiation.cc:299] Negotiation successful
0807 22:58:43.505283 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260807 22:58:43.514844 12538 raft_consensus.cc:1261] T 1b9460e145db47f983002febe388784e P 4b1638ba908c459d8eeb299a892bf252 [term 1 FOLLOWER]: Refusing update from remote peer ad8e3419f4a245e39ce61bcf0be910bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 22:58:43.514864 12272 raft_consensus.cc:1261] T 1b9460e145db47f983002febe388784e P d84a367949804c4fa7cc45104cf515d1 [term 1 FOLLOWER]: Refusing update from remote peer ad8e3419f4a245e39ce61bcf0be910bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 22:58:43.515913 12915 consensus_queue.cc:1035] T 1b9460e145db47f983002febe388784e P ad8e3419f4a245e39ce61bcf0be910bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "d84a367949804c4fa7cc45104cf515d1" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 46545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 22:58:43.516292 12921 consensus_queue.cc:1035] T 1b9460e145db47f983002febe388784e P ad8e3419f4a245e39ce61bcf0be910bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b1638ba908c459d8eeb299a892bf252" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 37509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 22:58:43.532364 12926 mvcc.cc:205] Tried to move back new op lower bound from 7316043872305700864 to 7316043871948914688. Current Snapshot: MvccSnapshot[applied={T|T < 7316043872305700864}]
I20260807 22:58:43.537325 12928 mvcc.cc:205] Tried to move back new op lower bound from 7316043872305700864 to 7316043871948914688. Current Snapshot: MvccSnapshot[applied={T|T < 7316043872305700864}]
I20260807 22:58:43.548483 11965 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42582:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 22:58:43.548914 11965 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42582:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 22:58:43.552661 11965 catalog_manager.cc:5782] T 00000000000000000000000000000000 P aa0561b0c48b4fcf8c7b4dc414279f91: Sending DeleteTablet for 3 replicas of tablet 1b9460e145db47f983002febe388784e
I20260807 22:58:43.553972 12502 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b9460e145db47f983002febe388784e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:58:43 UTC) from {username='kudu', principal='kudu/127.25.94.62@KRBTEST.COM'} at 127.0.0.1:50224
I20260807 22:58:43.554807 12748 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b9460e145db47f983002febe388784e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:58:43 UTC) from {username='kudu', principal='kudu/127.25.94.62@KRBTEST.COM'} at 127.0.0.1:38868
I20260807 22:58:43.555467 12238 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b9460e145db47f983002febe388784e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:58:43 UTC) from {username='kudu', principal='kudu/127.25.94.62@KRBTEST.COM'} at 127.0.0.1:49032
I20260807 22:58:43.555970 12943 tablet_replica.cc:330] T 1b9460e145db47f983002febe388784e P 4b1638ba908c459d8eeb299a892bf252: stopping tablet replica
I20260807 22:58:43.556478 12943 raft_consensus.cc:2229] T 1b9460e145db47f983002febe388784e P 4b1638ba908c459d8eeb299a892bf252 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 22:58:43.557403 12943 raft_consensus.cc:2258] T 1b9460e145db47f983002febe388784e P 4b1638ba908c459d8eeb299a892bf252 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 22:58:43.559170 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 12116
I20260807 22:58:43.569996 12945 tablet_replica.cc:330] T 1b9460e145db47f983002febe388784e P d84a367949804c4fa7cc45104cf515d1: stopping tablet replica
I20260807 22:58:43.570727 12945 raft_consensus.cc:2229] T 1b9460e145db47f983002febe388784e P d84a367949804c4fa7cc45104cf515d1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 22:58:43.571527 12945 raft_consensus.cc:2258] T 1b9460e145db47f983002febe388784e P d84a367949804c4fa7cc45104cf515d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 22:58:43.571643 12943 ts_tablet_manager.cc:1905] T 1b9460e145db47f983002febe388784e P 4b1638ba908c459d8eeb299a892bf252: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 22:58:43.573920 12945 ts_tablet_manager.cc:1905] T 1b9460e145db47f983002febe388784e P d84a367949804c4fa7cc45104cf515d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 22:58:43.581768 12943 ts_tablet_manager.cc:1918] T 1b9460e145db47f983002febe388784e P 4b1638ba908c459d8eeb299a892bf252: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 22:58:43.582231 12943 log.cc:1196] T 1b9460e145db47f983002febe388784e P 4b1638ba908c459d8eeb299a892bf252: Deleting WAL directory at /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786143361779964-25976-0/minicluster-data/ts-1/wal/wals/1b9460e145db47f983002febe388784e
I20260807 22:58:43.583124 12943 ts_tablet_manager.cc:1939] T 1b9460e145db47f983002febe388784e P 4b1638ba908c459d8eeb299a892bf252: Deleting consensus metadata
I20260807 22:58:43.585253 12945 ts_tablet_manager.cc:1918] T 1b9460e145db47f983002febe388784e P d84a367949804c4fa7cc45104cf515d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 22:58:43.585402 11947 catalog_manager.cc:4841] TS 4b1638ba908c459d8eeb299a892bf252 (127.25.94.2:37509): tablet 1b9460e145db47f983002febe388784e (table test-table [id=9355baf3dd764951a4bf64a6ed24e358]) successfully deleted
I20260807 22:58:43.585743 12945 log.cc:1196] T 1b9460e145db47f983002febe388784e P d84a367949804c4fa7cc45104cf515d1: Deleting WAL directory at /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786143361779964-25976-0/minicluster-data/ts-0/wal/wals/1b9460e145db47f983002febe388784e
I20260807 22:58:43.586783 12945 ts_tablet_manager.cc:1939] T 1b9460e145db47f983002febe388784e P d84a367949804c4fa7cc45104cf515d1: Deleting consensus metadata
I20260807 22:58:43.589357 11947 catalog_manager.cc:4841] TS d84a367949804c4fa7cc45104cf515d1 (127.25.94.1:46545): tablet 1b9460e145db47f983002febe388784e (table test-table [id=9355baf3dd764951a4bf64a6ed24e358]) successfully deleted
I20260807 22:58:43.589785 12944 tablet_replica.cc:330] T 1b9460e145db47f983002febe388784e P ad8e3419f4a245e39ce61bcf0be910bf: stopping tablet replica
I20260807 22:58:43.591005 12944 raft_consensus.cc:2229] T 1b9460e145db47f983002febe388784e P ad8e3419f4a245e39ce61bcf0be910bf [term 1 LEADER]: Raft consensus shutting down.
I20260807 22:58:43.591926 12944 raft_consensus.cc:2258] T 1b9460e145db47f983002febe388784e P ad8e3419f4a245e39ce61bcf0be910bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 22:58:43.594655 12944 ts_tablet_manager.cc:1905] T 1b9460e145db47f983002febe388784e P ad8e3419f4a245e39ce61bcf0be910bf: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 07 22:58:43 dist-test-slave-z8x9 krb5kdc[11135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143517, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.1@KRBTEST.COM
I20260807 22:58:43.604872 12944 ts_tablet_manager.cc:1918] T 1b9460e145db47f983002febe388784e P ad8e3419f4a245e39ce61bcf0be910bf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 22:58:43.605096 12944 log.cc:1196] T 1b9460e145db47f983002febe388784e P ad8e3419f4a245e39ce61bcf0be910bf: Deleting WAL directory at /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786143361779964-25976-0/minicluster-data/ts-2/wal/wals/1b9460e145db47f983002febe388784e
I20260807 22:58:43.605746 12944 ts_tablet_manager.cc:1939] T 1b9460e145db47f983002febe388784e P ad8e3419f4a245e39ce61bcf0be910bf: Deleting consensus metadata
I20260807 22:58:43.607623 12900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:58:43.560108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49042 (local address 127.25.94.1:46545)
0807 22:58:43.560311 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0807 22:58:43.560318 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 22:58:43.583972 (+ 23654us) server_negotiation.cc:407] Connection header received
0807 22:58:43.586194 (+ 2222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:58:43.586200 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:58:43.586458 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:58:43.586597 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:58:43.589386 (+ 2789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:58:43.590732 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:58:43.593153 (+ 2421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:58:43.593913 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:58:43.594045 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:58:43.599697 (+ 5652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:58:43.599711 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:58:43.599724 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:58:43.599775 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:58:43.603872 (+ 4097us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:58:43.604696 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:58:43.604704 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:58:43.604707 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:58:43.604866 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:58:43.605330 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:58:43.605337 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:58:43.605344 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:58:43.605863 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:58:43.606343 (+ 480us) server_negotiation.cc:1035] Waiting for connection context
0807 22:58:43.606813 (+ 470us) server_negotiation.cc:299] Negotiation successful
0807 22:58:43.607057 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260807 22:58:43.608075 11945 catalog_manager.cc:4841] TS ad8e3419f4a245e39ce61bcf0be910bf (127.25.94.3:40071): tablet 1b9460e145db47f983002febe388784e (table test-table [id=9355baf3dd764951a4bf64a6ed24e358]) successfully deleted
I20260807 22:58:43.609030 12337 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 22:58:43.702942 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 12116
I20260807 22:58:43.731317 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 12362
Aug 07 22:58:43 dist-test-slave-z8x9 krb5kdc[11135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143517, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.2@KRBTEST.COM
I20260807 22:58:43.748040 12895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:58:43.732236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50232 (local address 127.25.94.2:37509)
0807 22:58:43.732454 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0807 22:58:43.732468 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0807 22:58:43.732489 (+ 21us) server_negotiation.cc:407] Connection header received
0807 22:58:43.732563 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:58:43.732568 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:58:43.732654 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:58:43.732778 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:58:43.734168 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:58:43.735304 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:58:43.736714 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:58:43.737105 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:58:43.737496 (+ 391us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:58:43.741776 (+ 4280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:58:43.741790 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:58:43.741806 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:58:43.741845 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:58:43.744874 (+ 3029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:58:43.745767 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:58:43.745779 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:58:43.745782 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:58:43.745922 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:58:43.746359 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:58:43.746363 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:58:43.746365 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:58:43.746744 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:58:43.747069 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0807 22:58:43.747371 (+ 302us) server_negotiation.cc:299] Negotiation successful
0807 22:58:43.747598 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260807 22:58:43.749006 12616 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 22:58:43.841754 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 12362
I20260807 22:58:43.874316 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 12643
Aug 07 22:58:43 dist-test-slave-z8x9 krb5kdc[11135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143517, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.3@KRBTEST.COM
I20260807 22:58:43.891644 12898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:58:43.875284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38890 (local address 127.25.94.3:40071)
0807 22:58:43.875501 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0807 22:58:43.875506 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 22:58:43.875529 (+ 23us) server_negotiation.cc:407] Connection header received
0807 22:58:43.875638 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:58:43.875645 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:58:43.875731 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:58:43.875846 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:58:43.877377 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:58:43.878227 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:58:43.879558 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:58:43.880113 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:58:43.880257 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:58:43.885125 (+ 4868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:58:43.885137 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:58:43.885155 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:58:43.885248 (+ 93us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:58:43.888481 (+ 3233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:58:43.889375 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:58:43.889383 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:58:43.889385 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:58:43.889553 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:58:43.889977 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:58:43.889981 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:58:43.889984 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:58:43.890403 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:58:43.890758 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0807 22:58:43.891044 (+ 286us) server_negotiation.cc:299] Negotiation successful
0807 22:58:43.891276 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260807 22:58:43.892500 12862 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 22:58:44.018857 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 12643
I20260807 22:58:44.048491 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 11923
I20260807 22:58:44.049865 12024 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 22:58:44.169013 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 11923
I20260807 22:58:44.206405 25976 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260807 22:58:44.552592 25976 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260807 22:58:44.552872 25976 mini_ranger.cc:66] Stopping Ranger...
I20260807 22:58:44.899224 25976 mini_ranger.cc:68] Stopped Ranger
2026-08-07 22:58:44.899 UTC [11188] LOG: received smart shutdown request
2026-08-07 22:58:44.903 UTC [11188] LOG: background worker "logical replication launcher" (PID 11207) exited with exit code 1
2026-08-07 22:58:44.903 UTC [11198] LOG: shutting down
2026-08-07 22:58:45.089 UTC [11188] LOG: database system is shut down
2026-08-07T22:58:45Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (54879 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 22:58:45 dist-test-slave-z8x9 krb5kdc[12971](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 22:58:45 dist-test-slave-z8x9 krb5kdc[12971](info): set up 2 sockets
Aug 07 22:58:45 dist-test-slave-z8x9 krb5kdc[12971](info): commencing operation
krb5kdc: starting...
W20260807 22:58:46.616467 25976 mini_kdc.cc:121] Time spent starting KDC: real 1.345s user 0.000s sys 0.033s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/krb5kdc/test-user.keytab.
Aug 07 22:58:46 dist-test-slave-z8x9 krb5kdc[12971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143526, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T22:58:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T22:58:46Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.25.94.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.25.94.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.25.94.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/krb5kdc/rangeradmin_127.25.94.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.25.94.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/krb5kdc/rangeradmin_127.25.94.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.25.94.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.25.94.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.25.94.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/krb5kdc/rangerlookup_127.25.94.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.25.94.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/krb5kdc/rangerlookup_127.25.94.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.25.94.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.25.94.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.94.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/krb5kdc/HTTP_127.25.94.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.25.94.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/krb5kdc/HTTP_127.25.94.20@KRBTEST.COM.keytab.
I20260807 22:58:47.004757 25976 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-tasknHbun_/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/postgres -l logfile start
2026-08-07 22:58:49.736 UTC [13031] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-07 22:58:49.737 UTC [13031] LOG: listening on IPv4 address "127.25.94.20", port 45115
2026-08-07 22:58:49.740 UTC [13031] LOG: listening on Unix socket "/tmp/.s.PGSQL.45115"
2026-08-07 22:58:49.767 UTC [13038] LOG: database system was shut down at 2026-08-07 22:58:48 UTC
2026-08-07 22:58:49.773 UTC [13031] LOG: database system is ready to accept connections
I20260807 22:58:51.002941 25976 mini_postgres.cc:96] Postgres bound to 45115
2026-08-07 22:58:51.024 UTC [13050] FATAL: database "slave" does not exist
127.25.94.20:45115 - accepting connections
I20260807 22:58:51.025014 25976 mini_ranger.cc:161] Starting Ranger...
I20260807 22:58:51.054811 25976 mini_ranger.cc:84] Created miniranger Postgres user
I20260807 22:58:51.626283 25976 mini_ranger.cc:87] Created ranger Postgres database
I20260807 22:58:51.626426 25976 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/ranger-admin
2026-08-07 22:58:51,938 [I] DB FLAVOR :POSTGRES
2026-08-07 22:58:51,938 [I] --------- Verifying Ranger DB connection ---------
2026-08-07 22:58:51,938 [I] Checking connection..
2026-08-07 22:58:51,939 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknHbun_/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknHbun_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.94.20:45115/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-07 22:58:52,211 [I] Checking connection passed.
2026-08-07 22:58:52,212 [I] --------- Verifying version history table ---------
2026-08-07 22:58:52,212 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknHbun_/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknHbun_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.94.20:45115/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-07 22:58:52,495 [I] Table x_db_version_h does not exist in database ranger
2026-08-07 22:58:52,496 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknHbun_/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknHbun_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.94.20:45115/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-07 22:58:52,775 [I] Table x_db_version_h does not exist in database ranger
2026-08-07 22:58:52,775 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-07 22:58:52,775 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknHbun_/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknHbun_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.94.20:45115/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasknHbun_/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-07 22:58:53.040 UTC [13166] WARNING: there is no transaction in progress
2026-08-07 22:58:53,049 [I] create_dbversion_catalog.sql file imported successfully
2026-08-07 22:58:53,049 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknHbun_/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknHbun_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.94.20:45115/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-07 22:58:53,324 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-07 22:58:53,324 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-07 22:58:53,324 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknHbun_/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknHbun_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.94.20:45115/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-07 22:58:53,579 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknHbun_/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknHbun_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.94.20:45115/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-07 22:58:53,852 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknHbun_/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknHbun_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.94.20:45115/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-z8x9.c.gcp-upstream.internal','N') ;"
2026-08-07 22:58:54,120 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknHbun_/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknHbun_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.94.20:45115/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-07 22:58:54,405 [I] Table x_portal_user does not exist in database ranger
2026-08-07 22:58:54,405 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknHbun_/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknHbun_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.94.20:45115/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-07 22:58:54,674 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-07 22:58:54,675 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknHbun_/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknHbun_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.94.20:45115/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-07 22:58:54,945 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknHbun_/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknHbun_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.94.20:45115/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-07 22:58:55,232 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-07 22:58:55,233 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknHbun_/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknHbun_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.94.20:45115/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasknHbun_/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-07 22:58:56.316 UTC [13373] WARNING: there is no transaction in progress
2026-08-07 22:58:56.334 UTC [13373] WARNING: there is no transaction in progress
2026-08-07 22:58:56.352 UTC [13373] WARNING: there is no transaction in progress
2026-08-07 22:58:56.370 UTC [13373] WARNING: there is no transaction in progress
2026-08-07 22:58:56.391 UTC [13373] WARNING: there is no transaction in progress
2026-08-07 22:58:56.411 UTC [13373] WARNING: there is no transaction in progress
2026-08-07 22:58:56.473 UTC [13373] WARNING: there is no transaction in progress
2026-08-07 22:58:56.481 UTC [13373] WARNING: there is no transaction in progress
2026-08-07 22:58:56.498 UTC [13373] WARNING: there is no transaction in progress
2026-08-07 22:58:56.509 UTC [13373] WARNING: there is no transaction in progress
2026-08-07 22:58:56.519 UTC [13373] WARNING: there is no transaction in progress
2026-08-07 22:58:56.534 UTC [13373] WARNING: there is no transaction in progress
2026-08-07 22:58:56.544 UTC [13373] WARNING: there is no transaction in progress
2026-08-07 22:58:56.551 UTC [13373] WARNING: there is no transaction in progress
2026-08-07 22:58:56.558 UTC [13373] WARNING: there is no transaction in progress
2026-08-07 22:58:56.566 UTC [13373] WARNING: there is no transaction in progress
2026-08-07 22:58:57,345 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-07 22:58:57,346 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknHbun_/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknHbun_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.94.20:45115/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-07 22:58:57,611 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-07 22:58:57,611 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknHbun_/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknHbun_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.94.20:45115/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-07 22:58:57,889 [I] Table x_portal_user already exists in database 'ranger'
2026-08-07 22:58:57,890 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknHbun_/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknHbun_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.94.20:45115/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-07 22:58:58,173 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-07 22:58:58,174 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknHbun_/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknHbun_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.94.20:45115/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-07 22:58:58,456 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknHbun_/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknHbun_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.94.20:45115/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-07 22:58:58,726 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknHbun_/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknHbun_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.94.20:45115/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-z8x9.c.gcp-upstream.internal';"
2026-08-07 22:58:58,968 [I] CORE_DB_SCHEMA import status has been updated
2026-08-07 22:58:58,969 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknHbun_/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknHbun_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.94.20:45115/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-07 22:58:59,237 [I] DB_PATCHES have already been applied
I20260807 22:58:59.244021 25976 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/ranger-admin:/tmp/dist-test-tasknHbun_/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-tasknHbun_/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-tasknHbun_/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknHbun_/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260807 22:58:59.244127 25976 mini_ranger.cc:193] Using host: 127.25.94.20
I20260807 22:58:59.263584 25976 mini_ranger.cc:227] Ranger admin URL: http://127.25.94.20:59587
Aug 07, 2026 10:58:59 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 07, 2026 10:58:59 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 07, 2026 10:58:59 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/ranger-admin/ews/webapp
Aug 07, 2026 10:58:59 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 07, 2026 10:58:59 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 07, 2026 10:59:00 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/ranger-admin/ews/webapp].
Aug 07, 2026 10:59:00 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260807 22:59:00.739249 25976 mini_ranger.cc:160] Time spent starting Ranger: real 9.714s user 0.003s sys 0.051s
Aug 07, 2026 10:59:21 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 07, 2026 10:59:21 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 07, 2026 10:59:21 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 07, 2026 10:59:21 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 07, 2026 10:59:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 07, 2026 10:59:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 07, 2026 10:59:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 07, 2026 10:59:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 07, 2026 10:59:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 07, 2026 10:59:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 07, 2026 10:59:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 07, 2026 10:59:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 07, 2026 10:59:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 07, 2026 10:59:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 07, 2026 10:59:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 07, 2026 10:59:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 07, 2026 10:59:21 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 07, 2026 10:59:22 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260807 22:59:23.154518 25976 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.25.94.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.25.94.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.25.94.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/krb5kdc/rangerkms_127.25.94.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.25.94.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/krb5kdc/rangerkms_127.25.94.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.25.94.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.25.94.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.94.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/krb5kdc/HTTP_127.25.94.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.25.94.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/krb5kdc/HTTP_127.25.94.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 07 22:59:23 dist-test-slave-z8x9 krb5kdc[12971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143563, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260807 22:59:23.383508 25976 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260807 22:59:23.409744 25976 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260807 22:59:24.339934 25976 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260807 22:59:24.340059 25976 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/ranger-kms
I20260807 22:59:24.340107 25976 mini_ranger_kms.cc:227] Using postgres at 127.25.94.20:45115
2026-08-07 22:59:24,765 [I] DB FLAVOR :POSTGRES
2026-08-07 22:59:24,765 [I] --------- Verifying Ranger DB connection ---------
2026-08-07 22:59:24,765 [I] Checking connection
2026-08-07 22:59:24,765 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknHbun_/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.94.20:45115/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-07 22:59:25,021 [I] connection success
2026-08-07 22:59:25,022 [I] --------- Verifying Ranger DB tables ---------
2026-08-07 22:59:25,022 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-07 22:59:25,022 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknHbun_/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.94.20:45115/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-07 22:59:25,307 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-07 22:59:25,307 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-07 22:59:25,307 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-07 22:59:25,307 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknHbun_/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.94.20:45115/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-07 22:59:25,577 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260807 22:59:26.129786 25976 mini_ranger_kms.cc:310] Created kms service
I20260807 22:59:26.332495 25976 mini_ranger_kms.cc:326] Created kudu user
I20260807 22:59:26.380049 25976 mini_ranger_kms.cc:343] Created rangerkms user
Aug 07 22:59:26 dist-test-slave-z8x9 krb5kdc[12971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143563, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.25.94.20@KRBTEST.COM
I20260807 22:59:26.792042 25976 mini_ranger_kms.cc:384] Added ranger policy
I20260807 22:59:26.792392 25976 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/ranger-kms:/tmp/dist-test-tasknHbun_/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknHbun_/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-tasknHbun_/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260807 22:59:26.792434 25976 mini_ranger_kms.cc:240] Using host: 127.25.94.20
I20260807 22:59:26.812755 25976 mini_ranger_kms.cc:276] Ranger KMS PID: 13730
I20260807 22:59:26.812904 25976 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.25.94.20:33965
Aug 07, 2026 10:59:27 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 07, 2026 10:59:27 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 07, 2026 10:59:28 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 07, 2026 10:59:28 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 07, 2026 10:59:28 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/ranger-kms/ews/webapp].
Aug 07, 2026 10:59:28 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 07 22:59:29 dist-test-slave-z8x9 krb5kdc[12971](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786143569, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.25.94.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Aug 07 22:59:29 dist-test-slave-z8x9 krb5kdc[12971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143569, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.25.94.20@KRBTEST.COM for HTTP/127.25.94.20@KRBTEST.COM
I20260807 22:59:29.725764 25976 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.342s user 0.088s sys 0.364s
I20260807 22:59:29.726045 25976 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260807 22:59:29.726233 25976 mini_ranger_kms.cc:401] 127.25.94.20:33965/kms/v1/keys
Aug 07 22:59:30 dist-test-slave-z8x9 krb5kdc[12971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143569, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.25.94.20@KRBTEST.COM for HTTP/127.25.94.20@KRBTEST.COM
Aug 07 22:59:32 dist-test-slave-z8x9 krb5kdc[12971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143572, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.25.94.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.62.keytab.
Entry for principal kudu/127.25.94.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.62.keytab.
I20260807 22:59:32.589715 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.62:46333
--webserver_interface=127.25.94.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.62.keytab
--principal=kudu/127.25.94.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.20:44793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.94.62:46333
--ranger_config_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.25.94.20:33965/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:59:32.931567 13826 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:59:32.931887 13826 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:59:32.932056 13826 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:59:32.939457 13826 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 22:59:32.939574 13826 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:59:32.939668 13826 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:59:32.939779 13826 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 22:59:32.939868 13826 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 22:59:32.950230 13826 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:44793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/master-0/wal
--ranger_kms_url=127.25.94.20:33965/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.25.94.62:46333
--ranger_config_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.62.keytab
--principal=kudu/127.25.94.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.62:46333
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.94.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 22:59:32.951617 13826 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:59:32.952963 13826 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:59:32.965839 13840 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:59:32.965817 13842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:59:32.966049 13844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:59:32.966531 13826 server_base.cc:1034] running on GCE node
I20260807 22:59:32.967523 13826 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:59:32.969254 13826 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:59:32.970520 13826 hybrid_clock.cc:648] HybridClock initialized: now 1786143572970469 us; error 45 us; skew 500 ppm
I20260807 22:59:32.970911 13826 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 22:59:32 dist-test-slave-z8x9 krb5kdc[12971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143572, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:59:32.979985 13826 init.cc:377] Logged in from keytab as kudu/127.25.94.62@KRBTEST.COM (short username kudu)
I20260807 22:59:32.981752 13826 webserver.cc:460] Webserver started at http://127.25.94.62:39715/ using document root <none> and password file <none>
I20260807 22:59:32.982470 13826 fs_manager.cc:359] Metadata directory not provided
I20260807 22:59:32.982612 13826 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:59:32.982940 13826 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 07 22:59:32 dist-test-slave-z8x9 krb5kdc[12971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143572, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.62@KRBTEST.COM for HTTP/127.25.94.20@KRBTEST.COM
I20260807 22:59:33.051707 13826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/master-0/data/instance:
uuid: "3966b4aeaddc4b73862f071464bbe3d3"
format_stamp: "Formatted at 2026-08-07 22:59:33 on dist-test-slave-z8x9"
server_key: "29e84ffbb3227e2692227b4b56a0c11f"
server_key_iv: "42ad80e1e449b6e25b00d427af37bb37"
server_key_version: "kuduclusterkey@0"
I20260807 22:59:33.052640 13826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/master-0/wal/instance:
uuid: "3966b4aeaddc4b73862f071464bbe3d3"
format_stamp: "Formatted at 2026-08-07 22:59:33 on dist-test-slave-z8x9"
server_key: "29e84ffbb3227e2692227b4b56a0c11f"
server_key_iv: "42ad80e1e449b6e25b00d427af37bb37"
server_key_version: "kuduclusterkey@0"
I20260807 22:59:33.058576 13826 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260807 22:59:33.078728 13853 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:59:33.080504 13826 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260807 22:59:33.080709 13826 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/master-0/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/master-0/wal
uuid: "3966b4aeaddc4b73862f071464bbe3d3"
format_stamp: "Formatted at 2026-08-07 22:59:33 on dist-test-slave-z8x9"
server_key: "29e84ffbb3227e2692227b4b56a0c11f"
server_key_iv: "42ad80e1e449b6e25b00d427af37bb37"
server_key_version: "kuduclusterkey@0"
I20260807 22:59:33.081002 13826 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:59:33.096426 13826 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:59:33.101043 13826 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:59:33.101446 13826 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:59:33.123153 13826 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.62:46333
I20260807 22:59:33.123173 13951 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.62:46333 every 8 connection(s)
I20260807 22:59:33.124797 13826 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
I20260807 22:59:33.129037 13954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:59:33.131254 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 13826
I20260807 22:59:33.131503 25976 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/master-0/wal/instance
I20260807 22:59:33.132385 25976 external_mini_cluster.cc:1428] Setting key 03c265d19908540cb80851617c8aeb35
I20260807 22:59:33.145149 13954 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3966b4aeaddc4b73862f071464bbe3d3: Bootstrap starting.
I20260807 22:59:33.149060 13954 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3966b4aeaddc4b73862f071464bbe3d3: Neither blocks nor log segments found. Creating new log.
I20260807 22:59:33.150218 13954 log.cc:824] T 00000000000000000000000000000000 P 3966b4aeaddc4b73862f071464bbe3d3: Log is configured to *not* fsync() on all Append() calls
I20260807 22:59:33.154172 13954 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3966b4aeaddc4b73862f071464bbe3d3: No bootstrap required, opened a new log
Aug 07 22:59:33 dist-test-slave-z8x9 krb5kdc[12971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143572, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.62@KRBTEST.COM
I20260807 22:59:33.166904 13954 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3966b4aeaddc4b73862f071464bbe3d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3966b4aeaddc4b73862f071464bbe3d3" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 46333 } }
I20260807 22:59:33.167460 13954 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3966b4aeaddc4b73862f071464bbe3d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:59:33.167644 13954 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3966b4aeaddc4b73862f071464bbe3d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3966b4aeaddc4b73862f071464bbe3d3, State: Initialized, Role: FOLLOWER
I20260807 22:59:33.168295 13957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:33.134898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57582 (local address 127.25.94.62:46333)
0807 22:59:33.140983 (+ 6085us) server_negotiation.cc:206] Beginning negotiation
0807 22:59:33.141002 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0807 22:59:33.141055 (+ 53us) server_negotiation.cc:407] Connection header received
0807 22:59:33.143976 (+ 2921us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:59:33.144015 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:59:33.144616 (+ 601us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:59:33.145124 (+ 508us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:59:33.147000 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:59:33.149071 (+ 2071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:59:33.151160 (+ 2089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:59:33.152340 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:59:33.152599 (+ 259us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:59:33.157215 (+ 4616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:59:33.157230 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:59:33.157253 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:59:33.157324 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:59:33.161622 (+ 4298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:59:33.162928 (+ 1306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:59:33.162941 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:59:33.162952 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:59:33.163274 (+ 322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:59:33.164038 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:59:33.164050 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:59:33.164052 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:59:33.165135 (+ 1083us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:59:33.165719 (+ 584us) server_negotiation.cc:1035] Waiting for connection context
0807 22:59:33.166392 (+ 673us) server_negotiation.cc:299] Negotiation successful
0807 22:59:33.166944 (+ 552us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5792,"thread_start_us":5797,"threads_started":1}
I20260807 22:59:33.168416 13954 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3966b4aeaddc4b73862f071464bbe3d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3966b4aeaddc4b73862f071464bbe3d3" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 46333 } }
I20260807 22:59:33.168728 13954 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3966b4aeaddc4b73862f071464bbe3d3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 22:59:33.168901 13954 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3966b4aeaddc4b73862f071464bbe3d3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 22:59:33.169111 13954 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3966b4aeaddc4b73862f071464bbe3d3 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:59:33.172605 13954 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3966b4aeaddc4b73862f071464bbe3d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3966b4aeaddc4b73862f071464bbe3d3" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 46333 } }
I20260807 22:59:33.173242 13954 leader_election.cc:302] T 00000000000000000000000000000000 P 3966b4aeaddc4b73862f071464bbe3d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3966b4aeaddc4b73862f071464bbe3d3; no voters:
I20260807 22:59:33.173923 13954 leader_election.cc:288] T 00000000000000000000000000000000 P 3966b4aeaddc4b73862f071464bbe3d3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 22:59:33.174062 13960 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3966b4aeaddc4b73862f071464bbe3d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:59:33.176227 13954 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3966b4aeaddc4b73862f071464bbe3d3 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 22:59:33.181708 13960 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3966b4aeaddc4b73862f071464bbe3d3 [term 1 LEADER]: Becoming Leader. State: Replica: 3966b4aeaddc4b73862f071464bbe3d3, State: Running, Role: LEADER
I20260807 22:59:33.182639 13960 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3966b4aeaddc4b73862f071464bbe3d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3966b4aeaddc4b73862f071464bbe3d3" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 46333 } }
I20260807 22:59:33.185755 13972 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3966b4aeaddc4b73862f071464bbe3d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3966b4aeaddc4b73862f071464bbe3d3. Latest consensus state: current_term: 1 leader_uuid: "3966b4aeaddc4b73862f071464bbe3d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3966b4aeaddc4b73862f071464bbe3d3" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 46333 } } }
I20260807 22:59:33.186012 13965 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3966b4aeaddc4b73862f071464bbe3d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3966b4aeaddc4b73862f071464bbe3d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3966b4aeaddc4b73862f071464bbe3d3" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 46333 } } }
I20260807 22:59:33.187098 13972 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3966b4aeaddc4b73862f071464bbe3d3 [sys.catalog]: This master's current role is: LEADER
I20260807 22:59:33.187098 13965 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3966b4aeaddc4b73862f071464bbe3d3 [sys.catalog]: This master's current role is: LEADER
I20260807 22:59:33.204170 13984 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 22:59:33.211632 13984 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 22:59:33.212486 13954 ranger_client.cc:318] Using new properties file: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260807 22:59:33.227110 13984 catalog_manager.cc:1347] Generated new cluster ID: a1a45a0c478d408790fe3afd0e6c53e3
I20260807 22:59:33.227389 13984 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 22:59:33.270803 13984 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 22:59:33.273105 13984 catalog_manager.cc:1504] Loading token signing keys...
I20260807 22:59:33.285092 13984 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3966b4aeaddc4b73862f071464bbe3d3: Generated new TSK 0
I20260807 22:59:33.287298 13984 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.94.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.1.keytab.
Entry for principal kudu/127.25.94.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.1.keytab.
I20260807 22:59:34.097781 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.25.94.20:33965/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.1:0
--local_ip_for_outbound_sockets=127.25.94.1
--webserver_interface=127.25.94.1
--webserver_port=0
--tserver_master_addrs=127.25.94.62:46333
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.1.keytab
--principal=kudu/127.25.94.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.20:44793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 07 22:59:34 dist-test-slave-z8x9 krb5kdc[12971](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786143574, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260807 22:59:34.511879 14039 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:59:34.512187 14039 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:59:34.512346 14039 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:59:34.520267 14039 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:59:34.520393 14039 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:59:34.520569 14039 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.1
I20260807 22:59:34.531548 14039 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:44793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.25.94.20:33965/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.1.keytab
--principal=kudu/127.25.94.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.94.1
--webserver_port=0
--tserver_master_addrs=127.25.94.62:46333
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.1
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 22:59:34.532900 14039 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:59:34.534405 14039 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:59:34.551134 14054 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:59:34.551584 14053 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:59:34.557919 14057 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:59:34.566248 14039 server_base.cc:1034] running on GCE node
I20260807 22:59:34.567293 14039 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:59:34.568576 14039 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:59:34.569896 14039 hybrid_clock.cc:648] HybridClock initialized: now 1786143574569792 us; error 85 us; skew 500 ppm
I20260807 22:59:34.570420 14039 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 22:59:34 dist-test-slave-z8x9 krb5kdc[12971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143574, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:59:34.574967 14039 init.cc:377] Logged in from keytab as kudu/127.25.94.1@KRBTEST.COM (short username kudu)
I20260807 22:59:34.579897 14039 webserver.cc:460] Webserver started at http://127.25.94.1:46715/ using document root <none> and password file <none>
I20260807 22:59:34.580778 14039 fs_manager.cc:359] Metadata directory not provided
I20260807 22:59:34.580938 14039 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:59:34.581372 14039 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 07 22:59:34 dist-test-slave-z8x9 krb5kdc[12971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143574, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.1@KRBTEST.COM for HTTP/127.25.94.20@KRBTEST.COM
I20260807 22:59:34.600138 14039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/ts-0/data/instance:
uuid: "595157aa1d844fe7b30a673e79ec752a"
format_stamp: "Formatted at 2026-08-07 22:59:34 on dist-test-slave-z8x9"
server_key: "801c41a93f3617b837301f38e679f8a8"
server_key_iv: "08385ab7a468f9e6107067fe19a68851"
server_key_version: "kuduclusterkey@0"
I20260807 22:59:34.601783 14039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/ts-0/wal/instance:
uuid: "595157aa1d844fe7b30a673e79ec752a"
format_stamp: "Formatted at 2026-08-07 22:59:34 on dist-test-slave-z8x9"
server_key: "801c41a93f3617b837301f38e679f8a8"
server_key_iv: "08385ab7a468f9e6107067fe19a68851"
server_key_version: "kuduclusterkey@0"
I20260807 22:59:34.615541 14039 fs_manager.cc:691] Time spent creating directory manager: real 0.013s user 0.004s sys 0.007s
I20260807 22:59:34.649976 14068 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:59:34.655404 14039 fs_manager.cc:725] Time spent opening block manager: real 0.018s user 0.000s sys 0.012s
I20260807 22:59:34.655745 14039 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/ts-0/wal
uuid: "595157aa1d844fe7b30a673e79ec752a"
format_stamp: "Formatted at 2026-08-07 22:59:34 on dist-test-slave-z8x9"
server_key: "801c41a93f3617b837301f38e679f8a8"
server_key_iv: "08385ab7a468f9e6107067fe19a68851"
server_key_version: "kuduclusterkey@0"
I20260807 22:59:34.656132 14039 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:59:34.698837 14039 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:59:34.703223 14039 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:59:34.703581 14039 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:59:34.704698 14039 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:59:34.706346 14039 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:59:34.706451 14039 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:59:34.706615 14039 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:59:34.706707 14039 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:59:34.813613 14039 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.1:44253
I20260807 22:59:34.813630 14276 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.1:44253 every 8 connection(s)
I20260807 22:59:34.815337 14039 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/ts-0/data/info.pb
I20260807 22:59:34.816319 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 14039
I20260807 22:59:34.816571 25976 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/ts-0/wal/instance
I20260807 22:59:34.817476 25976 external_mini_cluster.cc:1428] Setting key aa366b83151c3d921d1a3512cc53d282
Aug 07 22:59:34 dist-test-slave-z8x9 krb5kdc[12971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143574, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.1@KRBTEST.COM for kudu/127.25.94.62@KRBTEST.COM
I20260807 22:59:34.854341 14279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:34.819410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:46333 (local address 127.25.94.1:36071)
0807 22:59:34.828523 (+ 9113us) negotiation.cc:107] Waiting for socket to connect
0807 22:59:34.828607 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0807 22:59:34.831118 (+ 2511us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:59:34.833610 (+ 2492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:59:34.833622 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:59:34.834338 (+ 716us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:59:34.835232 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:59:34.835260 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:59:34.836824 (+ 1564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:59:34.836830 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:59:34.838989 (+ 2159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:59:34.839006 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:59:34.840338 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:59:34.840347 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:59:34.840675 (+ 328us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:59:34.841621 (+ 946us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:59:34.841654 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:59:34.844727 (+ 3073us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:59:34.849279 (+ 4552us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:59:34.849290 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:59:34.849311 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:59:34.850003 (+ 692us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:59:34.850498 (+ 495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:59:34.850503 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:59:34.850507 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:59:34.850803 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:59:34.851498 (+ 695us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:59:34.851508 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:59:34.851961 (+ 453us) client_negotiation.cc:769] Sending connection context
0807 22:59:34.852287 (+ 326us) client_negotiation.cc:240] Negotiation successful
0807 22:59:34.852883 (+ 596us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8752,"thread_start_us":12056,"threads_started":1}
I20260807 22:59:34.855348 14281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:34.825429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.1:36071 (local address 127.25.94.62:46333)
0807 22:59:34.832799 (+ 7370us) server_negotiation.cc:206] Beginning negotiation
0807 22:59:34.832806 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 22:59:34.832828 (+ 22us) server_negotiation.cc:407] Connection header received
0807 22:59:34.832888 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:59:34.832893 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:59:34.832982 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:59:34.833102 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:59:34.835458 (+ 2356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:59:34.836577 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:59:34.839185 (+ 2608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:59:34.839870 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:59:34.840002 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:59:34.845051 (+ 5049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:59:34.845059 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:59:34.845064 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:59:34.845110 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:59:34.849078 (+ 3968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:59:34.850184 (+ 1106us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:59:34.850190 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:59:34.850194 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:59:34.850353 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:59:34.850949 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:59:34.850954 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:59:34.850958 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:59:34.851337 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:59:34.851493 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0807 22:59:34.854739 (+ 3246us) server_negotiation.cc:299] Negotiation successful
0807 22:59:34.854981 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7213,"thread_start_us":7082,"threads_started":1}
I20260807 22:59:34.857530 14277 heartbeater.cc:347] Connected to a master server at 127.25.94.62:46333
I20260807 22:59:34.857899 14277 heartbeater.cc:464] Registering TS with master...
I20260807 22:59:34.858801 14277 heartbeater.cc:511] Master 127.25.94.62:46333 requested a full tablet report, sending...
I20260807 22:59:34.861490 13884 ts_manager.cc:194] Registered new tserver with Master: 595157aa1d844fe7b30a673e79ec752a (127.25.94.1:44253)
I20260807 22:59:34.863603 13884 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.94.1@KRBTEST.COM'} at 127.25.94.1:36071
WARNING: no policy specified for kudu/127.25.94.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.2.keytab.
Entry for principal kudu/127.25.94.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.2.keytab.
I20260807 22:59:34.922153 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.25.94.20:33965/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.2:0
--local_ip_for_outbound_sockets=127.25.94.2
--webserver_interface=127.25.94.2
--webserver_port=0
--tserver_master_addrs=127.25.94.62:46333
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.2.keytab
--principal=kudu/127.25.94.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.20:44793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 07 22:59:34 dist-test-slave-z8x9 krb5kdc[12971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143574, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.62@KRBTEST.COM for HTTP/127.25.94.20@KRBTEST.COM
Aug 07 22:59:35 dist-test-slave-z8x9 krb5kdc[12971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143574, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.62@KRBTEST.COM for HTTP/127.25.94.20@KRBTEST.COM
W20260807 22:59:35.407379 14287 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:59:35.407819 14287 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:59:35.408041 14287 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:59:35.419286 14287 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:59:35.419531 14287 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:59:35.419759 14287 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.2
I20260807 22:59:35.433928 14287 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:44793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.25.94.20:33965/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.2.keytab
--principal=kudu/127.25.94.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.94.2
--webserver_port=0
--tserver_master_addrs=127.25.94.62:46333
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.2
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 22:59:35.436010 14287 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:59:35.437970 14287 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:59:35.464411 14297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:59:35.465401 14299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:59:35.467465 14302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:59:35.471501 14287 server_base.cc:1034] running on GCE node
I20260807 22:59:35.472632 14287 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:59:35.482292 14287 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:59:35.485795 14287 hybrid_clock.cc:648] HybridClock initialized: now 1786143575485704 us; error 87 us; skew 500 ppm
I20260807 22:59:35.486337 14287 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 22:59:35 dist-test-slave-z8x9 krb5kdc[12971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143575, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:59:35.490111 14287 init.cc:377] Logged in from keytab as kudu/127.25.94.2@KRBTEST.COM (short username kudu)
I20260807 22:59:35.496017 14287 webserver.cc:460] Webserver started at http://127.25.94.2:35031/ using document root <none> and password file <none>
I20260807 22:59:35.497025 14287 fs_manager.cc:359] Metadata directory not provided
I20260807 22:59:35.497231 14287 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:59:35.497589 14287 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 07 22:59:35 dist-test-slave-z8x9 krb5kdc[12971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143575, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.2@KRBTEST.COM for HTTP/127.25.94.20@KRBTEST.COM
I20260807 22:59:35.519930 14287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/ts-1/data/instance:
uuid: "5c352d1c025747bb86d7f33a5270c5dd"
format_stamp: "Formatted at 2026-08-07 22:59:35 on dist-test-slave-z8x9"
server_key: "3a993c6271f2ce5f893ad51bab40984f"
server_key_iv: "ca2a1f10c91f4955d98cec4def61fdc2"
server_key_version: "kuduclusterkey@0"
I20260807 22:59:35.521139 14287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/ts-1/wal/instance:
uuid: "5c352d1c025747bb86d7f33a5270c5dd"
format_stamp: "Formatted at 2026-08-07 22:59:35 on dist-test-slave-z8x9"
server_key: "3a993c6271f2ce5f893ad51bab40984f"
server_key_iv: "ca2a1f10c91f4955d98cec4def61fdc2"
server_key_version: "kuduclusterkey@0"
I20260807 22:59:35.541937 14287 fs_manager.cc:691] Time spent creating directory manager: real 0.020s user 0.014s sys 0.005s
I20260807 22:59:35.577288 14324 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:59:35.579483 14287 fs_manager.cc:725] Time spent opening block manager: real 0.016s user 0.002s sys 0.013s
I20260807 22:59:35.579689 14287 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/ts-1/wal
uuid: "5c352d1c025747bb86d7f33a5270c5dd"
format_stamp: "Formatted at 2026-08-07 22:59:35 on dist-test-slave-z8x9"
server_key: "3a993c6271f2ce5f893ad51bab40984f"
server_key_iv: "ca2a1f10c91f4955d98cec4def61fdc2"
server_key_version: "kuduclusterkey@0"
I20260807 22:59:35.579957 14287 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:59:35.614749 14287 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:59:35.620009 14287 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:59:35.620363 14287 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:59:35.621440 14287 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:59:35.622978 14287 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:59:35.623090 14287 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:59:35.623258 14287 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:59:35.623344 14287 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:59:35.664253 14287 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.2:41899
I20260807 22:59:35.664352 14520 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.2:41899 every 8 connection(s)
I20260807 22:59:35.665678 14287 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/ts-1/data/info.pb
I20260807 22:59:35.669860 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 14287
I20260807 22:59:35.670043 25976 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/ts-1/wal/instance
I20260807 22:59:35.670608 25976 external_mini_cluster.cc:1428] Setting key 10b316485bd8e475a310ff31816ab265
Aug 07 22:59:35 dist-test-slave-z8x9 krb5kdc[12971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143575, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.2@KRBTEST.COM for kudu/127.25.94.62@KRBTEST.COM
I20260807 22:59:35.689654 14528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:35.669309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:46333 (local address 127.25.94.2:58681)
0807 22:59:35.670165 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0807 22:59:35.670224 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0807 22:59:35.672301 (+ 2077us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:59:35.673123 (+ 822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:59:35.673131 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:59:35.673646 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:59:35.674250 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:59:35.674268 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:59:35.675333 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:59:35.675344 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:59:35.676672 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:59:35.676706 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:59:35.677635 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:59:35.677642 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:59:35.677939 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:59:35.678677 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:59:35.678704 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:59:35.680822 (+ 2118us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:59:35.684190 (+ 3368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:59:35.684198 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:59:35.684223 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:59:35.684979 (+ 756us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:59:35.685492 (+ 513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:59:35.685498 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:59:35.685502 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:59:35.685803 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:59:35.686763 (+ 960us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:59:35.686776 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:59:35.687209 (+ 433us) client_negotiation.cc:769] Sending connection context
0807 22:59:35.687563 (+ 354us) client_negotiation.cc:240] Negotiation successful
0807 22:59:35.688109 (+ 546us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"thread_start_us":360,"threads_started":1}
I20260807 22:59:35.690056 14524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:35.668699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.2:58681 (local address 127.25.94.62:46333)
0807 22:59:35.669288 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0807 22:59:35.669295 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 22:59:35.671338 (+ 2043us) server_negotiation.cc:407] Connection header received
0807 22:59:35.672620 (+ 1282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:59:35.672627 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:59:35.672737 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:59:35.672857 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:59:35.674435 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:59:35.675166 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:59:35.676845 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:59:35.677385 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:59:35.677530 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:59:35.681070 (+ 3540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:59:35.681075 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:59:35.681078 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:59:35.681109 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:59:35.684011 (+ 2902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:59:35.685097 (+ 1086us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:59:35.685103 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:59:35.685107 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:59:35.685317 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:59:35.685987 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:59:35.685993 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:59:35.685997 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:59:35.686469 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:59:35.686612 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0807 22:59:35.689411 (+ 2799us) server_negotiation.cc:299] Negotiation successful
0807 22:59:35.689675 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":263,"threads_started":1}
I20260807 22:59:35.691533 14522 heartbeater.cc:347] Connected to a master server at 127.25.94.62:46333
I20260807 22:59:35.691871 14522 heartbeater.cc:464] Registering TS with master...
I20260807 22:59:35.692821 14522 heartbeater.cc:511] Master 127.25.94.62:46333 requested a full tablet report, sending...
I20260807 22:59:35.694621 13884 ts_manager.cc:194] Registered new tserver with Master: 5c352d1c025747bb86d7f33a5270c5dd (127.25.94.2:41899)
I20260807 22:59:35.695752 13884 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.94.2@KRBTEST.COM'} at 127.25.94.2:58681
WARNING: no policy specified for kudu/127.25.94.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.3.keytab.
Entry for principal kudu/127.25.94.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.3.keytab.
I20260807 22:59:35.740275 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.25.94.20:33965/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.3:0
--local_ip_for_outbound_sockets=127.25.94.3
--webserver_interface=127.25.94.3
--webserver_port=0
--tserver_master_addrs=127.25.94.62:46333
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.3.keytab
--principal=kudu/127.25.94.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.20:44793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 22:59:35.867594 14277 heartbeater.cc:503] Master 127.25.94.62:46333 was elected leader, sending a full tablet report...
W20260807 22:59:36.105700 14531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:59:36.106022 14531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:59:36.106120 14531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:59:36.114500 14531 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:59:36.114862 14531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:59:36.115051 14531 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.3
I20260807 22:59:36.128094 14531 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:44793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.25.94.20:33965/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.3.keytab
--principal=kudu/127.25.94.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.94.3
--webserver_port=0
--tserver_master_addrs=127.25.94.62:46333
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.3
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 22:59:36.129858 14531 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:59:36.131611 14531 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:59:36.146664 14540 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:59:36.146715 14537 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:59:36.146683 14544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:59:36.150942 14531 server_base.cc:1034] running on GCE node
I20260807 22:59:36.151651 14531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:59:36.152757 14531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:59:36.154011 14531 hybrid_clock.cc:648] HybridClock initialized: now 1786143576153957 us; error 40 us; skew 500 ppm
I20260807 22:59:36.154449 14531 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 22:59:36 dist-test-slave-z8x9 krb5kdc[12971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143576, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:59:36.158125 14531 init.cc:377] Logged in from keytab as kudu/127.25.94.3@KRBTEST.COM (short username kudu)
I20260807 22:59:36.160141 14531 webserver.cc:460] Webserver started at http://127.25.94.3:40953/ using document root <none> and password file <none>
I20260807 22:59:36.160923 14531 fs_manager.cc:359] Metadata directory not provided
I20260807 22:59:36.161044 14531 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:59:36.161495 14531 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 07 22:59:36 dist-test-slave-z8x9 krb5kdc[12971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143576, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.3@KRBTEST.COM for HTTP/127.25.94.20@KRBTEST.COM
I20260807 22:59:36.179615 14531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/ts-2/data/instance:
uuid: "887cc7d8436e4c79bb7f1d36c89bc491"
format_stamp: "Formatted at 2026-08-07 22:59:36 on dist-test-slave-z8x9"
server_key: "c4c10e3ab8024364527fa5e1144d295a"
server_key_iv: "5650c1f9bba8651c47f98a7fcd542579"
server_key_version: "kuduclusterkey@0"
I20260807 22:59:36.180567 14531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/ts-2/wal/instance:
uuid: "887cc7d8436e4c79bb7f1d36c89bc491"
format_stamp: "Formatted at 2026-08-07 22:59:36 on dist-test-slave-z8x9"
server_key: "c4c10e3ab8024364527fa5e1144d295a"
server_key_iv: "5650c1f9bba8651c47f98a7fcd542579"
server_key_version: "kuduclusterkey@0"
I20260807 22:59:36.187449 14531 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260807 22:59:36.204663 14553 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:59:36.206319 14531 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260807 22:59:36.206553 14531 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/ts-2/wal
uuid: "887cc7d8436e4c79bb7f1d36c89bc491"
format_stamp: "Formatted at 2026-08-07 22:59:36 on dist-test-slave-z8x9"
server_key: "c4c10e3ab8024364527fa5e1144d295a"
server_key_iv: "5650c1f9bba8651c47f98a7fcd542579"
server_key_version: "kuduclusterkey@0"
I20260807 22:59:36.206806 14531 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:59:36.229101 14531 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:59:36.233455 14531 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:59:36.233793 14531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:59:36.234829 14531 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:59:36.236265 14531 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:59:36.236387 14531 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:59:36.236582 14531 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:59:36.236658 14531 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:59:36.273396 14531 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.3:43173
I20260807 22:59:36.273444 14758 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.3:43173 every 8 connection(s)
I20260807 22:59:36.274729 14531 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/ts-2/data/info.pb
I20260807 22:59:36.282187 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 14531
I20260807 22:59:36.282415 25976 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/ts-2/wal/instance
I20260807 22:59:36.282935 25976 external_mini_cluster.cc:1428] Setting key eeeb24109228694e78558fcb3e670370
Aug 07 22:59:36 dist-test-slave-z8x9 krb5kdc[12971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143576, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.3@KRBTEST.COM for kudu/127.25.94.62@KRBTEST.COM
I20260807 22:59:36.298072 14762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:36.277574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.3:50939 (local address 127.25.94.62:46333)
0807 22:59:36.278089 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0807 22:59:36.278095 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 22:59:36.279143 (+ 1048us) server_negotiation.cc:407] Connection header received
0807 22:59:36.280915 (+ 1772us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:59:36.280920 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:59:36.280995 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:59:36.281101 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:59:36.282864 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:59:36.283657 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:59:36.285886 (+ 2229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:59:36.286445 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:59:36.286550 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:59:36.290388 (+ 3838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:59:36.290394 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:59:36.290398 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:59:36.290437 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:59:36.293376 (+ 2939us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:59:36.294533 (+ 1157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:59:36.294539 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:59:36.294542 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:59:36.294742 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:59:36.295444 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:59:36.295449 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:59:36.295452 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:59:36.295943 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:59:36.296100 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0807 22:59:36.297462 (+ 1362us) server_negotiation.cc:299] Negotiation successful
0807 22:59:36.297698 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":245,"threads_started":1}
I20260807 22:59:36.298990 14766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:36.278200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:46333 (local address 127.25.94.3:50939)
0807 22:59:36.278960 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0807 22:59:36.279009 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0807 22:59:36.280623 (+ 1614us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:59:36.281437 (+ 814us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:59:36.281446 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:59:36.281981 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:59:36.282668 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:59:36.282689 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:59:36.283847 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:59:36.283851 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:59:36.285523 (+ 1672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:59:36.285538 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:59:36.286690 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:59:36.286717 (+ 27us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:59:36.286994 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:59:36.287788 (+ 794us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:59:36.287811 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:59:36.290134 (+ 2323us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:59:36.293575 (+ 3441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:59:36.293584 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:59:36.293600 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:59:36.294362 (+ 762us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:59:36.294874 (+ 512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:59:36.294879 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:59:36.294883 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:59:36.295261 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:59:36.296184 (+ 923us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:59:36.296193 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:59:36.296657 (+ 464us) client_negotiation.cc:769] Sending connection context
0807 22:59:36.297046 (+ 389us) client_negotiation.cc:240] Negotiation successful
0807 22:59:36.297672 (+ 626us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":314,"threads_started":1}
I20260807 22:59:36.300411 14759 heartbeater.cc:347] Connected to a master server at 127.25.94.62:46333
I20260807 22:59:36.300741 14759 heartbeater.cc:464] Registering TS with master...
I20260807 22:59:36.301533 14759 heartbeater.cc:511] Master 127.25.94.62:46333 requested a full tablet report, sending...
I20260807 22:59:36.303095 13884 ts_manager.cc:194] Registered new tserver with Master: 887cc7d8436e4c79bb7f1d36c89bc491 (127.25.94.3:43173)
I20260807 22:59:36.303969 13884 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.94.3@KRBTEST.COM'} at 127.25.94.3:50939
I20260807 22:59:36.311734 25976 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 22:59:36.313550 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 14039
Aug 07 22:59:36 dist-test-slave-z8x9 krb5kdc[12971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143572, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.1@KRBTEST.COM
I20260807 22:59:36.330317 14770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:36.314467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39254 (local address 127.25.94.1:44253)
0807 22:59:36.315101 (+ 634us) server_negotiation.cc:206] Beginning negotiation
0807 22:59:36.315108 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 22:59:36.315135 (+ 27us) server_negotiation.cc:407] Connection header received
0807 22:59:36.315243 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:59:36.315249 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:59:36.315475 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:59:36.315603 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:59:36.317017 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:59:36.318148 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:59:36.319643 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:59:36.320181 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:59:36.320281 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:59:36.324774 (+ 4493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:59:36.324787 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:59:36.324802 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:59:36.324843 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:59:36.327654 (+ 2811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:59:36.328346 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:59:36.328353 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:59:36.328355 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:59:36.328508 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:59:36.328828 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:59:36.328832 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:59:36.328833 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:59:36.329231 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:59:36.329515 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0807 22:59:36.329789 (+ 274us) server_negotiation.cc:299] Negotiation successful
0807 22:59:36.329999 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":468,"thread_start_us":305,"threads_started":1}
I20260807 22:59:36.331660 14271 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 22:59:36.415961 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 14039
I20260807 22:59:36.442595 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 14287
Aug 07 22:59:36 dist-test-slave-z8x9 krb5kdc[12971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143572, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.2@KRBTEST.COM
I20260807 22:59:36.464901 14775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:36.443619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52048 (local address 127.25.94.2:41899)
0807 22:59:36.444135 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0807 22:59:36.444142 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 22:59:36.444176 (+ 34us) server_negotiation.cc:407] Connection header received
0807 22:59:36.444251 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:59:36.444258 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:59:36.444454 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:59:36.444640 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:59:36.446055 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:59:36.447189 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:59:36.448595 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:59:36.449226 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:59:36.449397 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:59:36.458734 (+ 9337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:59:36.458752 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:59:36.458769 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:59:36.458816 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:59:36.461839 (+ 3023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:59:36.462633 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:59:36.462640 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:59:36.462642 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:59:36.462787 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:59:36.463221 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:59:36.463228 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:59:36.463236 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:59:36.463596 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:59:36.463926 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0807 22:59:36.464241 (+ 315us) server_negotiation.cc:299] Negotiation successful
0807 22:59:36.464524 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":274,"threads_started":1}
I20260807 22:59:36.466362 14512 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 22:59:36.549623 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 14287
I20260807 22:59:36.573419 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 14531
Aug 07 22:59:36 dist-test-slave-z8x9 krb5kdc[12971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143572, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.3@KRBTEST.COM
I20260807 22:59:36.590976 14777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:36.574468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45586 (local address 127.25.94.3:43173)
0807 22:59:36.575115 (+ 647us) server_negotiation.cc:206] Beginning negotiation
0807 22:59:36.575122 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 22:59:36.575149 (+ 27us) server_negotiation.cc:407] Connection header received
0807 22:59:36.575208 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:59:36.575215 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:59:36.575433 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:59:36.575607 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:59:36.577055 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:59:36.578208 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:59:36.579477 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:59:36.580056 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:59:36.580198 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:59:36.584700 (+ 4502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:59:36.584713 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:59:36.584728 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:59:36.584764 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:59:36.587917 (+ 3153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:59:36.588687 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:59:36.588700 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:59:36.588702 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:59:36.588832 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:59:36.589352 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:59:36.589356 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:59:36.589358 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:59:36.589701 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:59:36.590044 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0807 22:59:36.590349 (+ 305us) server_negotiation.cc:299] Negotiation successful
0807 22:59:36.590616 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":501,"thread_start_us":328,"threads_started":1}
I20260807 22:59:36.592226 14753 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 22:59:36.678890 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 14531
I20260807 22:59:36.703383 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 13826
I20260807 22:59:36.704808 13945 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 22:59:36.811398 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 13826
I20260807 22:59:36.843480 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.62:46333
--webserver_interface=127.25.94.62
--webserver_port=39715
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.62.keytab
--principal=kudu/127.25.94.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.20:44793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.94.62:46333
--ranger_config_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.25.94.20:33965/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.25.94.20:33965/kms with env {KRB5CCNAME=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:59:37.187062 14788 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:59:37.187393 14788 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:59:37.187562 14788 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:59:37.195673 14788 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 22:59:37.195811 14788 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:59:37.195896 14788 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:59:37.196007 14788 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 22:59:37.196091 14788 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 22:59:37.207719 14788 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:44793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.25.94.20:33965/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.25.94.62:46333
--ranger_config_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.62.keytab
--principal=kudu/127.25.94.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.62:46333
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.94.62
--webserver_port=39715
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 22:59:37.209369 14788 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:59:37.210964 14788 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:59:37.224380 14799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:59:37.224522 14801 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:59:37.224522 14804 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:59:37.225543 14788 server_base.cc:1034] running on GCE node
I20260807 22:59:37.226444 14788 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:59:37.228066 14788 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:59:37.229403 14788 hybrid_clock.cc:648] HybridClock initialized: now 1786143577229313 us; error 79 us; skew 500 ppm
I20260807 22:59:37.229848 14788 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 22:59:37 dist-test-slave-z8x9 krb5kdc[12971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143577, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:59:37.233804 14788 init.cc:377] Logged in from keytab as kudu/127.25.94.62@KRBTEST.COM (short username kudu)
I20260807 22:59:37.235472 14788 webserver.cc:460] Webserver started at http://127.25.94.62:39715/ using document root <none> and password file <none>
I20260807 22:59:37.236195 14788 fs_manager.cc:359] Metadata directory not provided
I20260807 22:59:37.236339 14788 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 07 22:59:37 dist-test-slave-z8x9 krb5kdc[12971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143577, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.62@KRBTEST.COM for HTTP/127.25.94.20@KRBTEST.COM
I20260807 22:59:37.284547 14788 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260807 22:59:37.287959 14818 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:59:37.289758 14788 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 22:59:37.289925 14788 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/master-0/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/master-0/wal
uuid: "3966b4aeaddc4b73862f071464bbe3d3"
format_stamp: "Formatted at 2026-08-07 22:59:33 on dist-test-slave-z8x9"
server_key: "29e84ffbb3227e2692227b4b56a0c11f"
server_key_iv: "42ad80e1e449b6e25b00d427af37bb37"
server_key_version: "kuduclusterkey@0"
I20260807 22:59:37.290877 14788 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:59:37.322335 14788 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:59:37.326586 14788 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:59:37.326848 14788 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:59:37.348138 14788 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.62:46333
I20260807 22:59:37.348157 14928 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.62:46333 every 8 connection(s)
I20260807 22:59:37.349932 14788 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
I20260807 22:59:37.352794 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 14788
I20260807 22:59:37.353927 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.25.94.20:33965/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.1:44253
--local_ip_for_outbound_sockets=127.25.94.1
--tserver_master_addrs=127.25.94.62:46333
--webserver_port=46715
--webserver_interface=127.25.94.1
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.1.keytab
--principal=kudu/127.25.94.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.20:44793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.25.94.20:33965/kms with env {KRB5CCNAME=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 22:59:37.357816 14932 sys_catalog.cc:263] Verifying existing consensus state
I20260807 22:59:37.361462 14932 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3966b4aeaddc4b73862f071464bbe3d3: Bootstrap starting.
I20260807 22:59:37.381876 14932 log.cc:824] T 00000000000000000000000000000000 P 3966b4aeaddc4b73862f071464bbe3d3: Log is configured to *not* fsync() on all Append() calls
I20260807 22:59:37.392093 14932 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3966b4aeaddc4b73862f071464bbe3d3: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 22:59:37.392941 14932 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3966b4aeaddc4b73862f071464bbe3d3: Bootstrap complete.
I20260807 22:59:37.402911 14932 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3966b4aeaddc4b73862f071464bbe3d3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3966b4aeaddc4b73862f071464bbe3d3" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 46333 } }
I20260807 22:59:37.403864 14932 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3966b4aeaddc4b73862f071464bbe3d3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3966b4aeaddc4b73862f071464bbe3d3, State: Initialized, Role: FOLLOWER
I20260807 22:59:37.404559 14932 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3966b4aeaddc4b73862f071464bbe3d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3966b4aeaddc4b73862f071464bbe3d3" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 46333 } }
I20260807 22:59:37.404829 14932 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3966b4aeaddc4b73862f071464bbe3d3 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 22:59:37.404975 14932 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3966b4aeaddc4b73862f071464bbe3d3 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 22:59:37.405160 14932 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3966b4aeaddc4b73862f071464bbe3d3 [term 1 FOLLOWER]: Advancing to term 2
I20260807 22:59:37.409104 14932 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3966b4aeaddc4b73862f071464bbe3d3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3966b4aeaddc4b73862f071464bbe3d3" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 46333 } }
I20260807 22:59:37.409682 14932 leader_election.cc:302] T 00000000000000000000000000000000 P 3966b4aeaddc4b73862f071464bbe3d3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3966b4aeaddc4b73862f071464bbe3d3; no voters:
I20260807 22:59:37.410378 14932 leader_election.cc:288] T 00000000000000000000000000000000 P 3966b4aeaddc4b73862f071464bbe3d3 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260807 22:59:37.410537 14941 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3966b4aeaddc4b73862f071464bbe3d3 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 22:59:37.411943 14941 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3966b4aeaddc4b73862f071464bbe3d3 [term 2 LEADER]: Becoming Leader. State: Replica: 3966b4aeaddc4b73862f071464bbe3d3, State: Running, Role: LEADER
I20260807 22:59:37.412612 14941 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3966b4aeaddc4b73862f071464bbe3d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3966b4aeaddc4b73862f071464bbe3d3" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 46333 } }
I20260807 22:59:37.412809 14932 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3966b4aeaddc4b73862f071464bbe3d3 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 22:59:37.415693 14943 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3966b4aeaddc4b73862f071464bbe3d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "3966b4aeaddc4b73862f071464bbe3d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3966b4aeaddc4b73862f071464bbe3d3" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 46333 } } }
I20260807 22:59:37.416016 14943 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3966b4aeaddc4b73862f071464bbe3d3 [sys.catalog]: This master's current role is: LEADER
I20260807 22:59:37.415268 14944 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3966b4aeaddc4b73862f071464bbe3d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3966b4aeaddc4b73862f071464bbe3d3. Latest consensus state: current_term: 2 leader_uuid: "3966b4aeaddc4b73862f071464bbe3d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3966b4aeaddc4b73862f071464bbe3d3" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 46333 } } }
I20260807 22:59:37.416610 14944 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3966b4aeaddc4b73862f071464bbe3d3 [sys.catalog]: This master's current role is: LEADER
I20260807 22:59:37.428606 14949 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 22:59:37.434999 14949 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 22:59:37.436689 14949 catalog_manager.cc:1259] Loaded cluster ID: a1a45a0c478d408790fe3afd0e6c53e3
I20260807 22:59:37.436915 14949 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 22:59:37.441401 14949 catalog_manager.cc:1504] Loading token signing keys...
I20260807 22:59:37.443010 14949 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 3966b4aeaddc4b73862f071464bbe3d3: Loaded TSK: 0
I20260807 22:59:37.444670 14949 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 22:59:37.450394 14932 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260807 22:59:37.825133 14936 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:59:37.825699 14936 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:59:37.825913 14936 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:59:37.838299 14936 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:59:37.838533 14936 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:59:37.838796 14936 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.1
I20260807 22:59:37.855593 14936 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:44793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.25.94.20:33965/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.1.keytab
--principal=kudu/127.25.94.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.1:44253
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.94.1
--webserver_port=46715
--tserver_master_addrs=127.25.94.62:46333
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.1
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 22:59:37.857429 14936 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:59:37.859272 14936 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:59:37.880769 15001 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:59:37.886415 15002 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:59:37.886718 14936 server_base.cc:1034] running on GCE node
W20260807 22:59:37.888036 15004 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:59:37.888968 14936 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:59:37.891039 14936 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:59:37.892400 14936 hybrid_clock.cc:648] HybridClock initialized: now 1786143577892288 us; error 67 us; skew 500 ppm
I20260807 22:59:37.892881 14936 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 22:59:37 dist-test-slave-z8x9 krb5kdc[12971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143577, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:59:37.897578 14936 init.cc:377] Logged in from keytab as kudu/127.25.94.1@KRBTEST.COM (short username kudu)
I20260807 22:59:37.905956 14936 webserver.cc:460] Webserver started at http://127.25.94.1:46715/ using document root <none> and password file <none>
I20260807 22:59:37.907037 14936 fs_manager.cc:359] Metadata directory not provided
I20260807 22:59:37.907217 14936 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 07 22:59:37 dist-test-slave-z8x9 krb5kdc[12971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143577, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.1@KRBTEST.COM for HTTP/127.25.94.20@KRBTEST.COM
I20260807 22:59:37.937881 14936 fs_manager.cc:709] Time spent opening directory manager: real 0.008s user 0.006s sys 0.001s
I20260807 22:59:37.940950 15020 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:59:37.942864 14936 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260807 22:59:37.943284 14936 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/ts-0/wal
uuid: "595157aa1d844fe7b30a673e79ec752a"
format_stamp: "Formatted at 2026-08-07 22:59:34 on dist-test-slave-z8x9"
server_key: "801c41a93f3617b837301f38e679f8a8"
server_key_iv: "08385ab7a468f9e6107067fe19a68851"
server_key_version: "kuduclusterkey@0"
I20260807 22:59:37.944890 14936 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:59:38.011199 14936 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:59:38.022909 14936 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:59:38.024713 14936 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:59:38.036444 14936 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:59:38.051481 14936 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:59:38.051728 14936 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260807 22:59:38.051942 14936 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:59:38.052085 14936 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:59:38.148346 14936 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.1:44253
I20260807 22:59:38.148435 15206 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.1:44253 every 8 connection(s)
I20260807 22:59:38.150609 14936 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/ts-0/data/info.pb
I20260807 22:59:38.155476 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 14936
I20260807 22:59:38.156172 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.25.94.20:33965/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.2:41899
--local_ip_for_outbound_sockets=127.25.94.2
--tserver_master_addrs=127.25.94.62:46333
--webserver_port=35031
--webserver_interface=127.25.94.2
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.2.keytab
--principal=kudu/127.25.94.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.20:44793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.25.94.20:33965/kms with env {KRB5CCNAME=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 07 22:59:38 dist-test-slave-z8x9 krb5kdc[12971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143577, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.1@KRBTEST.COM for kudu/127.25.94.62@KRBTEST.COM
I20260807 22:59:38.201584 15213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:38.154889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:46333 (local address 127.25.94.1:52029)
0807 22:59:38.166373 (+ 11484us) negotiation.cc:107] Waiting for socket to connect
0807 22:59:38.166457 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0807 22:59:38.171577 (+ 5120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:59:38.174914 (+ 3337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:59:38.174928 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:59:38.176785 (+ 1857us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:59:38.177813 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:59:38.177845 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:59:38.180722 (+ 2877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:59:38.180729 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:59:38.182957 (+ 2228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:59:38.182980 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:59:38.184793 (+ 1813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:59:38.184806 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:59:38.185162 (+ 356us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:59:38.186182 (+ 1020us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:59:38.186222 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:59:38.189526 (+ 3304us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:59:38.195237 (+ 5711us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:59:38.195249 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:59:38.195267 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:59:38.196048 (+ 781us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:59:38.196887 (+ 839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:59:38.196896 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:59:38.196903 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:59:38.197289 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:59:38.198599 (+ 1310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:59:38.198609 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:59:38.199052 (+ 443us) client_negotiation.cc:769] Sending connection context
0807 22:59:38.199403 (+ 351us) client_negotiation.cc:240] Negotiation successful
0807 22:59:38.200067 (+ 664us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10957,"thread_start_us":11850,"threads_started":1}
I20260807 22:59:38.204324 15215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:38.154872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.1:52029 (local address 127.25.94.62:46333)
0807 22:59:38.172800 (+ 17928us) server_negotiation.cc:206] Beginning negotiation
0807 22:59:38.172852 (+ 52us) server_negotiation.cc:399] Waiting for connection header
0807 22:59:38.172914 (+ 62us) server_negotiation.cc:407] Connection header received
0807 22:59:38.173514 (+ 600us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:59:38.173539 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:59:38.174101 (+ 562us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:59:38.174614 (+ 513us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:59:38.178185 (+ 3571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:59:38.180419 (+ 2234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:59:38.183278 (+ 2859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:59:38.184444 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:59:38.184694 (+ 250us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:59:38.190012 (+ 5318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:59:38.190028 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:59:38.190053 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:59:38.190118 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:59:38.195018 (+ 4900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:59:38.196273 (+ 1255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:59:38.196284 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:59:38.196293 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:59:38.196688 (+ 395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:59:38.197465 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:59:38.197473 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:59:38.197478 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:59:38.198269 (+ 791us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:59:38.198494 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0807 22:59:38.202432 (+ 3938us) server_negotiation.cc:299] Negotiation successful
0807 22:59:38.202928 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17618,"thread_start_us":17339,"threads_started":1}
I20260807 22:59:38.208511 15208 heartbeater.cc:347] Connected to a master server at 127.25.94.62:46333
I20260807 22:59:38.208879 15208 heartbeater.cc:464] Registering TS with master...
I20260807 22:59:38.209945 15208 heartbeater.cc:511] Master 127.25.94.62:46333 requested a full tablet report, sending...
W20260807 22:59:38.213873 15208 heartbeater.cc:644] Failed to heartbeat to 127.25.94.62:46333 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260807 22:59:38.214190 15208 heartbeater.cc:464] Registering TS with master...
I20260807 22:59:38.214532 15208 heartbeater.cc:511] Master 127.25.94.62:46333 requested a full tablet report, sending...
I20260807 22:59:38.215957 15208 heartbeater.cc:464] Registering TS with master...
I20260807 22:59:38.216253 15208 heartbeater.cc:511] Master 127.25.94.62:46333 requested a full tablet report, sending...
W20260807 22:59:38.217547 15208 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260807 22:59:38.593667 15216 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:59:38.594002 15216 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:59:38.594179 15216 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:59:38.602864 15216 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:59:38.603083 15216 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:59:38.603314 15216 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.2
I20260807 22:59:38.614111 15216 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:44793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.25.94.20:33965/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.2.keytab
--principal=kudu/127.25.94.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.2:41899
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.94.2
--webserver_port=35031
--tserver_master_addrs=127.25.94.62:46333
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.2
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 22:59:38.615569 15216 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:59:38.616921 15216 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:59:38.636009 15250 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:59:38.637224 15253 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:59:38.642688 15261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:59:38.643139 15216 server_base.cc:1034] running on GCE node
I20260807 22:59:38.644080 15216 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:59:38.645604 15216 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:59:38.646893 15216 hybrid_clock.cc:648] HybridClock initialized: now 1786143578646813 us; error 71 us; skew 500 ppm
I20260807 22:59:38.647354 15216 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 22:59:38 dist-test-slave-z8x9 krb5kdc[12971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143578, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:59:38.651877 15216 init.cc:377] Logged in from keytab as kudu/127.25.94.2@KRBTEST.COM (short username kudu)
I20260807 22:59:38.657871 15216 webserver.cc:460] Webserver started at http://127.25.94.2:35031/ using document root <none> and password file <none>
I20260807 22:59:38.658900 15216 fs_manager.cc:359] Metadata directory not provided
I20260807 22:59:38.659066 15216 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 07 22:59:38 dist-test-slave-z8x9 krb5kdc[12971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143578, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.2@KRBTEST.COM for HTTP/127.25.94.20@KRBTEST.COM
I20260807 22:59:38.719347 15216 fs_manager.cc:709] Time spent opening directory manager: real 0.015s user 0.011s sys 0.003s
I20260807 22:59:38.734328 15279 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:59:38.736502 15216 fs_manager.cc:725] Time spent opening block manager: real 0.015s user 0.005s sys 0.008s
I20260807 22:59:38.736831 15216 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/ts-1/wal
uuid: "5c352d1c025747bb86d7f33a5270c5dd"
format_stamp: "Formatted at 2026-08-07 22:59:35 on dist-test-slave-z8x9"
server_key: "3a993c6271f2ce5f893ad51bab40984f"
server_key_iv: "ca2a1f10c91f4955d98cec4def61fdc2"
server_key_version: "kuduclusterkey@0"
I20260807 22:59:38.738565 15216 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:59:38.770063 15216 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:59:38.775537 15216 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:59:38.775930 15216 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:59:38.777379 15216 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:59:38.779359 15216 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:59:38.779515 15216 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260807 22:59:38.779666 15216 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:59:38.779791 15216 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 07 22:59:38 dist-test-slave-z8x9 krb5kdc[12971](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786143578, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:59:38.861425 15216 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.2:41899
I20260807 22:59:38.861496 15463 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.2:41899 every 8 connection(s)
I20260807 22:59:38.863377 15216 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/ts-1/data/info.pb
I20260807 22:59:38.867698 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 15216
I20260807 22:59:38.868502 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.25.94.20:33965/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.3:43173
--local_ip_for_outbound_sockets=127.25.94.3
--tserver_master_addrs=127.25.94.62:46333
--webserver_port=40953
--webserver_interface=127.25.94.3
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.3.keytab
--principal=kudu/127.25.94.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.20:44793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.25.94.20:33965/kms with env {KRB5CCNAME=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 07 22:59:38 dist-test-slave-z8x9 krb5kdc[12971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143578, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.2@KRBTEST.COM for kudu/127.25.94.62@KRBTEST.COM
I20260807 22:59:38.899518 15467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:38.866816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:46333 (local address 127.25.94.2:38595)
0807 22:59:38.873059 (+ 6243us) negotiation.cc:107] Waiting for socket to connect
0807 22:59:38.873109 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0807 22:59:38.875610 (+ 2501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:59:38.876581 (+ 971us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:59:38.876597 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:59:38.877339 (+ 742us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:59:38.878202 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:59:38.878231 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:59:38.879958 (+ 1727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:59:38.879967 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:59:38.882938 (+ 2971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:59:38.882959 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:59:38.884081 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:59:38.884091 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:59:38.884409 (+ 318us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:59:38.886310 (+ 1901us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:59:38.886353 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:59:38.889637 (+ 3284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:59:38.894010 (+ 4373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:59:38.894022 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:59:38.894041 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:59:38.894831 (+ 790us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:59:38.895318 (+ 487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:59:38.895323 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:59:38.895326 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:59:38.895743 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:59:38.896594 (+ 851us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:59:38.896602 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:59:38.897070 (+ 468us) client_negotiation.cc:769] Sending connection context
0807 22:59:38.897429 (+ 359us) client_negotiation.cc:240] Negotiation successful
0807 22:59:38.898046 (+ 617us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5944,"thread_start_us":5726,"threads_started":1}
I20260807 22:59:38.900678 15472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:38.868979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.2:38595 (local address 127.25.94.62:46333)
0807 22:59:38.872462 (+ 3483us) server_negotiation.cc:206] Beginning negotiation
0807 22:59:38.872473 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 22:59:38.873605 (+ 1132us) server_negotiation.cc:407] Connection header received
0807 22:59:38.876001 (+ 2396us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:59:38.876010 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:59:38.876115 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:59:38.876294 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:59:38.878416 (+ 2122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:59:38.879736 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:59:38.883135 (+ 3399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:59:38.883793 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:59:38.883936 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:59:38.889958 (+ 6022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:59:38.889966 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:59:38.889971 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:59:38.890017 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:59:38.893790 (+ 3773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:59:38.895014 (+ 1224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:59:38.895020 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:59:38.895024 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:59:38.895178 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:59:38.895895 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:59:38.895901 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:59:38.895905 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:59:38.896363 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:59:38.896514 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0807 22:59:38.900054 (+ 3540us) server_negotiation.cc:299] Negotiation successful
0807 22:59:38.900301 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3288,"thread_start_us":3183,"threads_started":1}
I20260807 22:59:38.903018 15464 heartbeater.cc:347] Connected to a master server at 127.25.94.62:46333
I20260807 22:59:38.903380 15464 heartbeater.cc:464] Registering TS with master...
I20260807 22:59:38.904345 15464 heartbeater.cc:511] Master 127.25.94.62:46333 requested a full tablet report, sending...
I20260807 22:59:38.906833 14854 ts_manager.cc:194] Registered new tserver with Master: 5c352d1c025747bb86d7f33a5270c5dd (127.25.94.2:41899)
I20260807 22:59:38.909027 14854 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.94.2@KRBTEST.COM'} at 127.25.94.2:38595
I20260807 22:59:39.217732 15208 heartbeater.cc:464] Registering TS with master...
I20260807 22:59:39.218226 15208 heartbeater.cc:511] Master 127.25.94.62:46333 requested a full tablet report, sending...
I20260807 22:59:39.219787 14854 ts_manager.cc:194] Registered new tserver with Master: 595157aa1d844fe7b30a673e79ec752a (127.25.94.1:44253)
I20260807 22:59:39.221138 14854 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.94.1@KRBTEST.COM'} at 127.25.94.1:52029
Aug 07 22:59:39 dist-test-slave-z8x9 krb5kdc[12971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143578, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.62@KRBTEST.COM for HTTP/127.25.94.20@KRBTEST.COM
W20260807 22:59:39.328572 15473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:59:39.328933 15473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:59:39.329118 15473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:59:39.336823 15473 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:59:39.337007 15473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:59:39.337284 15473 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.3
I20260807 22:59:39.348587 15473 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:44793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.25.94.20:33965/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.3.keytab
--principal=kudu/127.25.94.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.3:43173
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.94.3
--webserver_port=40953
--tserver_master_addrs=127.25.94.62:46333
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.3
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 22:59:39.350406 15473 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:59:39.351980 15473 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:59:39.373864 15482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:59:39.374596 15481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:59:39.381358 15485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:59:39.382679 15473 server_base.cc:1034] running on GCE node
I20260807 22:59:39.383483 15473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:59:39.384694 15473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:59:39.385980 15473 hybrid_clock.cc:648] HybridClock initialized: now 1786143579385907 us; error 69 us; skew 500 ppm
I20260807 22:59:39.386344 15473 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 22:59:39 dist-test-slave-z8x9 krb5kdc[12971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143579, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:59:39.390759 15473 init.cc:377] Logged in from keytab as kudu/127.25.94.3@KRBTEST.COM (short username kudu)
I20260807 22:59:39.393064 15473 webserver.cc:460] Webserver started at http://127.25.94.3:40953/ using document root <none> and password file <none>
I20260807 22:59:39.394099 15473 fs_manager.cc:359] Metadata directory not provided
I20260807 22:59:39.394274 15473 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 07 22:59:39 dist-test-slave-z8x9 krb5kdc[12971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143579, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.3@KRBTEST.COM for HTTP/127.25.94.20@KRBTEST.COM
I20260807 22:59:39.418476 15473 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.002s sys 0.001s
I20260807 22:59:39.422279 15498 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:59:39.424132 15473 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260807 22:59:39.424361 15473 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/ts-2/wal
uuid: "887cc7d8436e4c79bb7f1d36c89bc491"
format_stamp: "Formatted at 2026-08-07 22:59:36 on dist-test-slave-z8x9"
server_key: "c4c10e3ab8024364527fa5e1144d295a"
server_key_iv: "5650c1f9bba8651c47f98a7fcd542579"
server_key_version: "kuduclusterkey@0"
I20260807 22:59:39.425765 15473 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:59:39.453502 15473 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:59:39.459036 15473 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:59:39.459362 15473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:59:39.460587 15473 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:59:39.462666 15473 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:59:39.462790 15473 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:59:39.462947 15473 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:59:39.463027 15473 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:59:39.553745 15473 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.3:43173
I20260807 22:59:39.553848 15697 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.3:43173 every 8 connection(s)
I20260807 22:59:39.555909 15473 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/ts-2/data/info.pb
Aug 07 22:59:39 dist-test-slave-z8x9 krb5kdc[12971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143578, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.62@KRBTEST.COM for HTTP/127.25.94.20@KRBTEST.COM
I20260807 22:59:39.563943 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 15473
Aug 07 22:59:39 dist-test-slave-z8x9 krb5kdc[12971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143579, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.3@KRBTEST.COM for kudu/127.25.94.62@KRBTEST.COM
I20260807 22:59:39.618575 15703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:39.561723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.3:53261 (local address 127.25.94.62:46333)
0807 22:59:39.582867 (+ 21144us) server_negotiation.cc:206] Beginning negotiation
0807 22:59:39.582878 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 22:59:39.589258 (+ 6380us) server_negotiation.cc:407] Connection header received
0807 22:59:39.594514 (+ 5256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:59:39.594521 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:59:39.594623 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:59:39.594716 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:59:39.597215 (+ 2499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:59:39.598362 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:59:39.600677 (+ 2315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:59:39.601283 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:59:39.601385 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:59:39.606049 (+ 4664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:59:39.606056 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:59:39.606060 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:59:39.606108 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:59:39.610052 (+ 3944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:59:39.611262 (+ 1210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:59:39.611268 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:59:39.611272 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:59:39.611428 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:59:39.615451 (+ 4023us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:59:39.615457 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:59:39.615461 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:59:39.615926 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:59:39.616096 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0807 22:59:39.617917 (+ 1821us) server_negotiation.cc:299] Negotiation successful
0807 22:59:39.618198 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20959,"spinlock_wait_cycles":4864,"thread_start_us":24362,"threads_started":1}
I20260807 22:59:39.624900 15700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:39.560964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:46333 (local address 127.25.94.3:53261)
0807 22:59:39.588575 (+ 27611us) negotiation.cc:107] Waiting for socket to connect
0807 22:59:39.588632 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0807 22:59:39.594081 (+ 5449us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:59:39.595388 (+ 1307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:59:39.595403 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:59:39.596070 (+ 667us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:59:39.596952 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:59:39.596980 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:59:39.598583 (+ 1603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:59:39.598588 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:59:39.600490 (+ 1902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:59:39.600507 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:59:39.601776 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:59:39.601783 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:59:39.602062 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:59:39.602922 (+ 860us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:59:39.602959 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:59:39.605722 (+ 2763us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:59:39.610311 (+ 4589us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:59:39.610320 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:59:39.610346 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:59:39.611054 (+ 708us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:59:39.611600 (+ 546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:59:39.611605 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:59:39.611608 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:59:39.611936 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:59:39.616203 (+ 4267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:59:39.616212 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:59:39.617044 (+ 832us) client_negotiation.cc:769] Sending connection context
0807 22:59:39.622770 (+ 5726us) client_negotiation.cc:240] Negotiation successful
0807 22:59:39.623432 (+ 662us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27266,"thread_start_us":26970,"threads_started":1}
I20260807 22:59:39.626075 15701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:39.565508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57596 (local address 127.25.94.62:46333)
0807 22:59:39.586453 (+ 20945us) server_negotiation.cc:206] Beginning negotiation
0807 22:59:39.586462 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 22:59:39.586492 (+ 30us) server_negotiation.cc:407] Connection header received
0807 22:59:39.586585 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:59:39.586594 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:59:39.586735 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:59:39.586899 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:59:39.603310 (+ 16411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:59:39.604512 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:59:39.608765 (+ 4253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:59:39.610289 (+ 1524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:59:39.610430 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:59:39.615432 (+ 5002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:59:39.615441 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:59:39.615446 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:59:39.615532 (+ 86us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:59:39.619972 (+ 4440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:59:39.620915 (+ 943us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:59:39.620921 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:59:39.620924 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:59:39.621089 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:59:39.621621 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:59:39.621626 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:59:39.621628 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:59:39.621984 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:59:39.622241 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0807 22:59:39.625479 (+ 3238us) server_negotiation.cc:299] Negotiation successful
0807 22:59:39.625718 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20763,"spinlock_wait_cycles":36992,"thread_start_us":17064,"threads_started":1}
I20260807 22:59:39.629743 15698 heartbeater.cc:347] Connected to a master server at 127.25.94.62:46333
I20260807 22:59:39.630112 15698 heartbeater.cc:464] Registering TS with master...
I20260807 22:59:39.631111 15698 heartbeater.cc:511] Master 127.25.94.62:46333 requested a full tablet report, sending...
I20260807 22:59:39.634158 14854 ts_manager.cc:194] Registered new tserver with Master: 887cc7d8436e4c79bb7f1d36c89bc491 (127.25.94.3:43173)
I20260807 22:59:39.635377 14854 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.94.3@KRBTEST.COM'} at 127.25.94.3:53261
I20260807 22:59:39.647722 25976 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 22:59:39.708935 15701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:39.652955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57608 (local address 127.25.94.62:46333)
0807 22:59:39.653190 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0807 22:59:39.653200 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 22:59:39.681838 (+ 28638us) server_negotiation.cc:407] Connection header received
0807 22:59:39.682214 (+ 376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:59:39.682221 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:59:39.682339 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:59:39.682429 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:59:39.686685 (+ 4256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:59:39.687882 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:59:39.692500 (+ 4618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:59:39.693210 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:59:39.693320 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:59:39.699069 (+ 5749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:59:39.699077 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:59:39.699082 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:59:39.699132 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:59:39.702749 (+ 3617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:59:39.703667 (+ 918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:59:39.703673 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:59:39.703677 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:59:39.703828 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:59:39.704392 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:59:39.704398 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:59:39.704401 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:59:39.704817 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:59:39.705076 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0807 22:59:39.708297 (+ 3221us) server_negotiation.cc:299] Negotiation successful
0807 22:59:39.708570 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260807 22:59:39.722128 14854 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57608:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 22:59:39.735083 14854 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 22:59:39.810705 15721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:39.787853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45594 (local address 127.25.94.3:43173)
0807 22:59:39.800205 (+ 12352us) server_negotiation.cc:206] Beginning negotiation
0807 22:59:39.800213 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 22:59:39.800248 (+ 35us) server_negotiation.cc:407] Connection header received
0807 22:59:39.800396 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:59:39.800405 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:59:39.800689 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:59:39.800906 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:59:39.802155 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:59:39.804000 (+ 1845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:59:39.807423 (+ 3423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:59:39.808968 (+ 1545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:59:39.809889 (+ 921us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:59:39.810095 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0807 22:59:39.810190 (+ 95us) server_negotiation.cc:299] Negotiation successful
0807 22:59:39.810366 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12179,"thread_start_us":12034,"threads_started":1}
I20260807 22:59:39.812173 15729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:39.789588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.1:44253 (local address 127.0.0.1:39262)
0807 22:59:39.801676 (+ 12088us) negotiation.cc:107] Waiting for socket to connect
0807 22:59:39.801697 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 22:59:39.801862 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:59:39.803384 (+ 1522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:59:39.803389 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:59:39.803417 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:59:39.803794 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:59:39.803814 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:59:39.805899 (+ 2085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:59:39.805908 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:59:39.808042 (+ 2134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:59:39.808058 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:59:39.811411 (+ 3353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:59:39.811417 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:59:39.811581 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:59:39.811613 (+ 32us) client_negotiation.cc:769] Sending connection context
0807 22:59:39.811704 (+ 91us) client_negotiation.cc:240] Negotiation successful
0807 22:59:39.811831 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11933,"thread_start_us":14572,"threads_started":1}
I20260807 22:59:39.813237 15727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:39.789798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39262 (local address 127.25.94.1:44253)
0807 22:59:39.799705 (+ 9907us) server_negotiation.cc:206] Beginning negotiation
0807 22:59:39.799715 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 22:59:39.802017 (+ 2302us) server_negotiation.cc:407] Connection header received
0807 22:59:39.802130 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:59:39.802137 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:59:39.802387 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:59:39.802594 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:59:39.804018 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:59:39.805703 (+ 1685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:59:39.808228 (+ 2525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:59:39.809659 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:59:39.809791 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:59:39.810047 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0807 22:59:39.812586 (+ 2539us) server_negotiation.cc:299] Negotiation successful
0807 22:59:39.812786 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9703,"thread_start_us":9553,"threads_started":1}
I20260807 22:59:39.812201 15720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:39.787215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.3:43173 (local address 127.0.0.1:45594)
0807 22:59:39.793026 (+ 5811us) negotiation.cc:107] Waiting for socket to connect
0807 22:59:39.793078 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0807 22:59:39.793305 (+ 227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:59:39.801319 (+ 8014us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:59:39.801331 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:59:39.801376 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:59:39.801914 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:59:39.801931 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:59:39.804157 (+ 2226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:59:39.804164 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:59:39.806835 (+ 2671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:59:39.806851 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:59:39.809228 (+ 2377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:59:39.809236 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:59:39.809477 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:59:39.809542 (+ 65us) client_negotiation.cc:769] Sending connection context
0807 22:59:39.809679 (+ 137us) client_negotiation.cc:240] Negotiation successful
0807 22:59:39.811739 (+ 2060us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5502,"mutex_wait_us":1801,"thread_start_us":5563,"threads_started":1}
I20260807 22:59:39.817400 15732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:39.794609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52058 (local address 127.25.94.2:41899)
0807 22:59:39.803295 (+ 8686us) server_negotiation.cc:206] Beginning negotiation
0807 22:59:39.803303 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 22:59:39.804793 (+ 1490us) server_negotiation.cc:407] Connection header received
0807 22:59:39.804897 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:59:39.804905 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:59:39.805123 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:59:39.805379 (+ 256us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:59:39.806348 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:59:39.809093 (+ 2745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:59:39.812293 (+ 3200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:59:39.815409 (+ 3116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:59:39.815546 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:59:39.815734 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0807 22:59:39.816871 (+ 1137us) server_negotiation.cc:299] Negotiation successful
0807 22:59:39.817045 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8534,"thread_start_us":8422,"threads_started":1}
I20260807 22:59:39.817785 15724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:39.793981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.2:41899 (local address 127.0.0.1:52058)
0807 22:59:39.804514 (+ 10533us) negotiation.cc:107] Waiting for socket to connect
0807 22:59:39.804533 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 22:59:39.804662 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:59:39.805627 (+ 965us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:59:39.805632 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:59:39.805658 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:59:39.806119 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:59:39.806135 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:59:39.809311 (+ 3176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:59:39.809317 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:59:39.811489 (+ 2172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:59:39.811506 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:59:39.815917 (+ 4411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:59:39.815925 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:59:39.816116 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:59:39.816154 (+ 38us) client_negotiation.cc:769] Sending connection context
0807 22:59:39.816287 (+ 133us) client_negotiation.cc:240] Negotiation successful
0807 22:59:39.816433 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10398,"mutex_wait_us":60,"thread_start_us":8663,"threads_started":1}
I20260807 22:59:39.818984 15591 tablet_service.cc:1467] Processing CreateTablet for tablet f80dafa100b04f76b5bdf5aae6283da7 (DEFAULT_TABLE table=test-table [id=5f5cd2d339d24c8e84e3a1db6d8fb193]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:59:39.819319 15112 tablet_service.cc:1467] Processing CreateTablet for tablet f80dafa100b04f76b5bdf5aae6283da7 (DEFAULT_TABLE table=test-table [id=5f5cd2d339d24c8e84e3a1db6d8fb193]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:59:39.820737 15591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f80dafa100b04f76b5bdf5aae6283da7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:59:39.820755 15112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f80dafa100b04f76b5bdf5aae6283da7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:59:39.821331 15350 tablet_service.cc:1467] Processing CreateTablet for tablet f80dafa100b04f76b5bdf5aae6283da7 (DEFAULT_TABLE table=test-table [id=5f5cd2d339d24c8e84e3a1db6d8fb193]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:59:39.822695 15350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f80dafa100b04f76b5bdf5aae6283da7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:59:39.851009 15734 tablet_bootstrap.cc:492] T f80dafa100b04f76b5bdf5aae6283da7 P 595157aa1d844fe7b30a673e79ec752a: Bootstrap starting.
I20260807 22:59:39.855630 15733 tablet_bootstrap.cc:492] T f80dafa100b04f76b5bdf5aae6283da7 P 5c352d1c025747bb86d7f33a5270c5dd: Bootstrap starting.
I20260807 22:59:39.856616 15735 tablet_bootstrap.cc:492] T f80dafa100b04f76b5bdf5aae6283da7 P 887cc7d8436e4c79bb7f1d36c89bc491: Bootstrap starting.
I20260807 22:59:39.857080 15734 tablet_bootstrap.cc:654] T f80dafa100b04f76b5bdf5aae6283da7 P 595157aa1d844fe7b30a673e79ec752a: Neither blocks nor log segments found. Creating new log.
I20260807 22:59:39.860052 15734 log.cc:824] T f80dafa100b04f76b5bdf5aae6283da7 P 595157aa1d844fe7b30a673e79ec752a: Log is configured to *not* fsync() on all Append() calls
I20260807 22:59:39.860509 15735 tablet_bootstrap.cc:654] T f80dafa100b04f76b5bdf5aae6283da7 P 887cc7d8436e4c79bb7f1d36c89bc491: Neither blocks nor log segments found. Creating new log.
I20260807 22:59:39.862149 15735 log.cc:824] T f80dafa100b04f76b5bdf5aae6283da7 P 887cc7d8436e4c79bb7f1d36c89bc491: Log is configured to *not* fsync() on all Append() calls
I20260807 22:59:39.870852 15734 tablet_bootstrap.cc:492] T f80dafa100b04f76b5bdf5aae6283da7 P 595157aa1d844fe7b30a673e79ec752a: No bootstrap required, opened a new log
I20260807 22:59:39.871364 15734 ts_tablet_manager.cc:1397] T f80dafa100b04f76b5bdf5aae6283da7 P 595157aa1d844fe7b30a673e79ec752a: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.012s
I20260807 22:59:39.873950 15733 tablet_bootstrap.cc:654] T f80dafa100b04f76b5bdf5aae6283da7 P 5c352d1c025747bb86d7f33a5270c5dd: Neither blocks nor log segments found. Creating new log.
I20260807 22:59:39.875406 15733 log.cc:824] T f80dafa100b04f76b5bdf5aae6283da7 P 5c352d1c025747bb86d7f33a5270c5dd: Log is configured to *not* fsync() on all Append() calls
I20260807 22:59:39.878690 15735 tablet_bootstrap.cc:492] T f80dafa100b04f76b5bdf5aae6283da7 P 887cc7d8436e4c79bb7f1d36c89bc491: No bootstrap required, opened a new log
I20260807 22:59:39.879053 15735 ts_tablet_manager.cc:1397] T f80dafa100b04f76b5bdf5aae6283da7 P 887cc7d8436e4c79bb7f1d36c89bc491: Time spent bootstrapping tablet: real 0.023s user 0.009s sys 0.011s
I20260807 22:59:39.883155 15734 raft_consensus.cc:348] T f80dafa100b04f76b5bdf5aae6283da7 P 595157aa1d844fe7b30a673e79ec752a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "887cc7d8436e4c79bb7f1d36c89bc491" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 43173 } } peers { permanent_uuid: "5c352d1c025747bb86d7f33a5270c5dd" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 41899 } } peers { permanent_uuid: "595157aa1d844fe7b30a673e79ec752a" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44253 } }
I20260807 22:59:39.883674 15734 raft_consensus.cc:374] T f80dafa100b04f76b5bdf5aae6283da7 P 595157aa1d844fe7b30a673e79ec752a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:59:39.883823 15734 raft_consensus.cc:729] T f80dafa100b04f76b5bdf5aae6283da7 P 595157aa1d844fe7b30a673e79ec752a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 595157aa1d844fe7b30a673e79ec752a, State: Initialized, Role: FOLLOWER
I20260807 22:59:39.884613 15734 consensus_queue.cc:260] T f80dafa100b04f76b5bdf5aae6283da7 P 595157aa1d844fe7b30a673e79ec752a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "887cc7d8436e4c79bb7f1d36c89bc491" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 43173 } } peers { permanent_uuid: "5c352d1c025747bb86d7f33a5270c5dd" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 41899 } } peers { permanent_uuid: "595157aa1d844fe7b30a673e79ec752a" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44253 } }
I20260807 22:59:39.888289 15733 tablet_bootstrap.cc:492] T f80dafa100b04f76b5bdf5aae6283da7 P 5c352d1c025747bb86d7f33a5270c5dd: No bootstrap required, opened a new log
I20260807 22:59:39.888608 15733 ts_tablet_manager.cc:1397] T f80dafa100b04f76b5bdf5aae6283da7 P 5c352d1c025747bb86d7f33a5270c5dd: Time spent bootstrapping tablet: real 0.033s user 0.010s sys 0.000s
I20260807 22:59:39.893800 15735 raft_consensus.cc:348] T f80dafa100b04f76b5bdf5aae6283da7 P 887cc7d8436e4c79bb7f1d36c89bc491 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "887cc7d8436e4c79bb7f1d36c89bc491" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 43173 } } peers { permanent_uuid: "5c352d1c025747bb86d7f33a5270c5dd" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 41899 } } peers { permanent_uuid: "595157aa1d844fe7b30a673e79ec752a" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44253 } }
I20260807 22:59:39.896339 15735 raft_consensus.cc:374] T f80dafa100b04f76b5bdf5aae6283da7 P 887cc7d8436e4c79bb7f1d36c89bc491 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:59:39.896550 15735 raft_consensus.cc:729] T f80dafa100b04f76b5bdf5aae6283da7 P 887cc7d8436e4c79bb7f1d36c89bc491 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 887cc7d8436e4c79bb7f1d36c89bc491, State: Initialized, Role: FOLLOWER
I20260807 22:59:39.897688 15735 consensus_queue.cc:260] T f80dafa100b04f76b5bdf5aae6283da7 P 887cc7d8436e4c79bb7f1d36c89bc491 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "887cc7d8436e4c79bb7f1d36c89bc491" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 43173 } } peers { permanent_uuid: "5c352d1c025747bb86d7f33a5270c5dd" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 41899 } } peers { permanent_uuid: "595157aa1d844fe7b30a673e79ec752a" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44253 } }
I20260807 22:59:39.899144 15208 heartbeater.cc:503] Master 127.25.94.62:46333 was elected leader, sending a full tablet report...
I20260807 22:59:39.899070 15733 raft_consensus.cc:348] T f80dafa100b04f76b5bdf5aae6283da7 P 5c352d1c025747bb86d7f33a5270c5dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "887cc7d8436e4c79bb7f1d36c89bc491" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 43173 } } peers { permanent_uuid: "5c352d1c025747bb86d7f33a5270c5dd" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 41899 } } peers { permanent_uuid: "595157aa1d844fe7b30a673e79ec752a" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44253 } }
I20260807 22:59:39.899562 15733 raft_consensus.cc:374] T f80dafa100b04f76b5bdf5aae6283da7 P 5c352d1c025747bb86d7f33a5270c5dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:59:39.899655 15733 raft_consensus.cc:729] T f80dafa100b04f76b5bdf5aae6283da7 P 5c352d1c025747bb86d7f33a5270c5dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c352d1c025747bb86d7f33a5270c5dd, State: Initialized, Role: FOLLOWER
I20260807 22:59:39.899739 15734 ts_tablet_manager.cc:1428] T f80dafa100b04f76b5bdf5aae6283da7 P 595157aa1d844fe7b30a673e79ec752a: Time spent starting tablet: real 0.028s user 0.013s sys 0.013s
I20260807 22:59:39.900367 15733 consensus_queue.cc:260] T f80dafa100b04f76b5bdf5aae6283da7 P 5c352d1c025747bb86d7f33a5270c5dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "887cc7d8436e4c79bb7f1d36c89bc491" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 43173 } } peers { permanent_uuid: "5c352d1c025747bb86d7f33a5270c5dd" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 41899 } } peers { permanent_uuid: "595157aa1d844fe7b30a673e79ec752a" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44253 } }
I20260807 22:59:39.904759 15735 ts_tablet_manager.cc:1428] T f80dafa100b04f76b5bdf5aae6283da7 P 887cc7d8436e4c79bb7f1d36c89bc491: Time spent starting tablet: real 0.026s user 0.016s sys 0.002s
I20260807 22:59:39.905251 15698 heartbeater.cc:503] Master 127.25.94.62:46333 was elected leader, sending a full tablet report...
W20260807 22:59:39.905536 15211 tablet.cc:2369] T f80dafa100b04f76b5bdf5aae6283da7 P 595157aa1d844fe7b30a673e79ec752a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 22:59:39.907887 15464 heartbeater.cc:503] Master 127.25.94.62:46333 was elected leader, sending a full tablet report...
I20260807 22:59:39.908454 15733 ts_tablet_manager.cc:1428] T f80dafa100b04f76b5bdf5aae6283da7 P 5c352d1c025747bb86d7f33a5270c5dd: Time spent starting tablet: real 0.020s user 0.007s sys 0.012s
I20260807 22:59:40.021837 15756 raft_consensus.cc:482] T f80dafa100b04f76b5bdf5aae6283da7 P 5c352d1c025747bb86d7f33a5270c5dd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 22:59:40.022193 15756 raft_consensus.cc:504] T f80dafa100b04f76b5bdf5aae6283da7 P 5c352d1c025747bb86d7f33a5270c5dd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "887cc7d8436e4c79bb7f1d36c89bc491" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 43173 } } peers { permanent_uuid: "5c352d1c025747bb86d7f33a5270c5dd" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 41899 } } peers { permanent_uuid: "595157aa1d844fe7b30a673e79ec752a" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44253 } }
I20260807 22:59:40.024091 15756 leader_election.cc:288] T f80dafa100b04f76b5bdf5aae6283da7 P 5c352d1c025747bb86d7f33a5270c5dd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 887cc7d8436e4c79bb7f1d36c89bc491 (127.25.94.3:43173), 595157aa1d844fe7b30a673e79ec752a (127.25.94.1:44253)
I20260807 22:59:40.032409 15758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:40.024505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.1:44253 (local address 127.25.94.2:56935)
0807 22:59:40.025327 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0807 22:59:40.025343 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 22:59:40.025443 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:59:40.026167 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:59:40.026172 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:59:40.026211 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:59:40.026657 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:59:40.026668 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:59:40.027962 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:59:40.027973 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:59:40.029850 (+ 1877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:59:40.029864 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:59:40.031540 (+ 1676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:59:40.031547 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:59:40.031719 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:59:40.031758 (+ 39us) client_negotiation.cc:769] Sending connection context
0807 22:59:40.031863 (+ 105us) client_negotiation.cc:240] Negotiation successful
0807 22:59:40.031998 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":715,"mutex_wait_us":93,"thread_start_us":1002,"threads_started":1}
I20260807 22:59:40.032611 15721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:40.025119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.2:57025 (local address 127.25.94.3:43173)
0807 22:59:40.025325 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0807 22:59:40.025331 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 22:59:40.025354 (+ 23us) server_negotiation.cc:407] Connection header received
0807 22:59:40.025467 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:59:40.025472 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:59:40.025547 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:59:40.025651 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:59:40.026492 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:59:40.027533 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:59:40.030159 (+ 2626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:59:40.031412 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:59:40.031526 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:59:40.031625 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0807 22:59:40.032231 (+ 606us) server_negotiation.cc:299] Negotiation successful
0807 22:59:40.032360 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260807 22:59:40.033381 15757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:40.024291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.3:43173 (local address 127.25.94.2:57025)
0807 22:59:40.025115 (+ 824us) negotiation.cc:107] Waiting for socket to connect
0807 22:59:40.025141 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0807 22:59:40.025318 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:59:40.025810 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:59:40.025815 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:59:40.025844 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:59:40.026323 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:59:40.026336 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:59:40.027698 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:59:40.027705 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:59:40.029997 (+ 2292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:59:40.030013 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:59:40.031811 (+ 1798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:59:40.031816 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:59:40.032035 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:59:40.032068 (+ 33us) client_negotiation.cc:769] Sending connection context
0807 22:59:40.032800 (+ 732us) client_negotiation.cc:240] Negotiation successful
0807 22:59:40.032942 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":644,"mutex_wait_us":49,"thread_start_us":420,"threads_started":1}
I20260807 22:59:40.033742 15727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:40.025604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.2:56935 (local address 127.25.94.1:44253)
0807 22:59:40.025837 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0807 22:59:40.025842 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 22:59:40.025859 (+ 17us) server_negotiation.cc:407] Connection header received
0807 22:59:40.025931 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:59:40.025939 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:59:40.026017 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:59:40.026121 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:59:40.026842 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:59:40.027817 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:59:40.029996 (+ 2179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:59:40.031214 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:59:40.031422 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:59:40.031607 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0807 22:59:40.031921 (+ 314us) server_negotiation.cc:299] Negotiation successful
0807 22:59:40.032062 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260807 22:59:40.034538 15617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f80dafa100b04f76b5bdf5aae6283da7" candidate_uuid: "5c352d1c025747bb86d7f33a5270c5dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "887cc7d8436e4c79bb7f1d36c89bc491" is_pre_election: true
I20260807 22:59:40.034674 15142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f80dafa100b04f76b5bdf5aae6283da7" candidate_uuid: "5c352d1c025747bb86d7f33a5270c5dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "595157aa1d844fe7b30a673e79ec752a" is_pre_election: true
I20260807 22:59:40.035094 15617 raft_consensus.cc:2454] T f80dafa100b04f76b5bdf5aae6283da7 P 887cc7d8436e4c79bb7f1d36c89bc491 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c352d1c025747bb86d7f33a5270c5dd in term 0.
I20260807 22:59:40.035200 15142 raft_consensus.cc:2454] T f80dafa100b04f76b5bdf5aae6283da7 P 595157aa1d844fe7b30a673e79ec752a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c352d1c025747bb86d7f33a5270c5dd in term 0.
I20260807 22:59:40.035792 15281 leader_election.cc:302] T f80dafa100b04f76b5bdf5aae6283da7 P 5c352d1c025747bb86d7f33a5270c5dd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 595157aa1d844fe7b30a673e79ec752a, 5c352d1c025747bb86d7f33a5270c5dd, 887cc7d8436e4c79bb7f1d36c89bc491; no voters:
I20260807 22:59:40.036234 15756 raft_consensus.cc:2781] T f80dafa100b04f76b5bdf5aae6283da7 P 5c352d1c025747bb86d7f33a5270c5dd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 22:59:40.036389 15756 raft_consensus.cc:482] T f80dafa100b04f76b5bdf5aae6283da7 P 5c352d1c025747bb86d7f33a5270c5dd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 22:59:40.036499 15756 raft_consensus.cc:3037] T f80dafa100b04f76b5bdf5aae6283da7 P 5c352d1c025747bb86d7f33a5270c5dd [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:59:40.038807 15756 raft_consensus.cc:504] T f80dafa100b04f76b5bdf5aae6283da7 P 5c352d1c025747bb86d7f33a5270c5dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "887cc7d8436e4c79bb7f1d36c89bc491" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 43173 } } peers { permanent_uuid: "5c352d1c025747bb86d7f33a5270c5dd" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 41899 } } peers { permanent_uuid: "595157aa1d844fe7b30a673e79ec752a" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44253 } }
I20260807 22:59:40.039520 15756 leader_election.cc:288] T f80dafa100b04f76b5bdf5aae6283da7 P 5c352d1c025747bb86d7f33a5270c5dd [CANDIDATE]: Term 1 election: Requested vote from peers 887cc7d8436e4c79bb7f1d36c89bc491 (127.25.94.3:43173), 595157aa1d844fe7b30a673e79ec752a (127.25.94.1:44253)
I20260807 22:59:40.039880 15617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f80dafa100b04f76b5bdf5aae6283da7" candidate_uuid: "5c352d1c025747bb86d7f33a5270c5dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "887cc7d8436e4c79bb7f1d36c89bc491"
I20260807 22:59:40.039965 15142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f80dafa100b04f76b5bdf5aae6283da7" candidate_uuid: "5c352d1c025747bb86d7f33a5270c5dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "595157aa1d844fe7b30a673e79ec752a"
I20260807 22:59:40.040087 15617 raft_consensus.cc:3037] T f80dafa100b04f76b5bdf5aae6283da7 P 887cc7d8436e4c79bb7f1d36c89bc491 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:59:40.040216 15142 raft_consensus.cc:3037] T f80dafa100b04f76b5bdf5aae6283da7 P 595157aa1d844fe7b30a673e79ec752a [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:59:40.042515 15142 raft_consensus.cc:2454] T f80dafa100b04f76b5bdf5aae6283da7 P 595157aa1d844fe7b30a673e79ec752a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c352d1c025747bb86d7f33a5270c5dd in term 1.
I20260807 22:59:40.042968 15280 leader_election.cc:302] T f80dafa100b04f76b5bdf5aae6283da7 P 5c352d1c025747bb86d7f33a5270c5dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 595157aa1d844fe7b30a673e79ec752a, 5c352d1c025747bb86d7f33a5270c5dd; no voters:
I20260807 22:59:40.043176 15617 raft_consensus.cc:2454] T f80dafa100b04f76b5bdf5aae6283da7 P 887cc7d8436e4c79bb7f1d36c89bc491 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c352d1c025747bb86d7f33a5270c5dd in term 1.
I20260807 22:59:40.043287 15756 raft_consensus.cc:2781] T f80dafa100b04f76b5bdf5aae6283da7 P 5c352d1c025747bb86d7f33a5270c5dd [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:59:40.043854 15756 raft_consensus.cc:686] T f80dafa100b04f76b5bdf5aae6283da7 P 5c352d1c025747bb86d7f33a5270c5dd [term 1 LEADER]: Becoming Leader. State: Replica: 5c352d1c025747bb86d7f33a5270c5dd, State: Running, Role: LEADER
I20260807 22:59:40.044427 15756 consensus_queue.cc:237] T f80dafa100b04f76b5bdf5aae6283da7 P 5c352d1c025747bb86d7f33a5270c5dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "887cc7d8436e4c79bb7f1d36c89bc491" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 43173 } } peers { permanent_uuid: "5c352d1c025747bb86d7f33a5270c5dd" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 41899 } } peers { permanent_uuid: "595157aa1d844fe7b30a673e79ec752a" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44253 } }
I20260807 22:59:40.049739 14854 catalog_manager.cc:5495] T f80dafa100b04f76b5bdf5aae6283da7 P 5c352d1c025747bb86d7f33a5270c5dd reported cstate change: term changed from 0 to 1, leader changed from <none> to 5c352d1c025747bb86d7f33a5270c5dd (127.25.94.2). New cstate: current_term: 1 leader_uuid: "5c352d1c025747bb86d7f33a5270c5dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "887cc7d8436e4c79bb7f1d36c89bc491" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 43173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5c352d1c025747bb86d7f33a5270c5dd" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 41899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "595157aa1d844fe7b30a673e79ec752a" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44253 } health_report { overall_health: UNKNOWN } } }
W20260807 22:59:40.073630 15699 tablet.cc:2369] T f80dafa100b04f76b5bdf5aae6283da7 P 887cc7d8436e4c79bb7f1d36c89bc491: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 22:59:40.105813 15732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:40.099202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52074 (local address 127.25.94.2:41899)
0807 22:59:40.099451 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0807 22:59:40.099460 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 22:59:40.099483 (+ 23us) server_negotiation.cc:407] Connection header received
0807 22:59:40.099599 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:59:40.099602 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:59:40.099680 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:59:40.099846 (+ 166us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 22:59:40.100259 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:59:40.101390 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:59:40.103440 (+ 2050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:59:40.103852 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:59:40.103955 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:59:40.104587 (+ 632us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 22:59:40.105123 (+ 536us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 22:59:40.105324 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0807 22:59:40.105504 (+ 180us) server_negotiation.cc:299] Negotiation successful
0807 22:59:40.105581 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260807 22:59:40.116708 15142 raft_consensus.cc:1261] T f80dafa100b04f76b5bdf5aae6283da7 P 595157aa1d844fe7b30a673e79ec752a [term 1 FOLLOWER]: Refusing update from remote peer 5c352d1c025747bb86d7f33a5270c5dd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 22:59:40.116708 15617 raft_consensus.cc:1261] T f80dafa100b04f76b5bdf5aae6283da7 P 887cc7d8436e4c79bb7f1d36c89bc491 [term 1 FOLLOWER]: Refusing update from remote peer 5c352d1c025747bb86d7f33a5270c5dd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 22:59:40.117962 15759 consensus_queue.cc:1035] T f80dafa100b04f76b5bdf5aae6283da7 P 5c352d1c025747bb86d7f33a5270c5dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "887cc7d8436e4c79bb7f1d36c89bc491" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 43173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 22:59:40.118330 15756 consensus_queue.cc:1035] T f80dafa100b04f76b5bdf5aae6283da7 P 5c352d1c025747bb86d7f33a5270c5dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "595157aa1d844fe7b30a673e79ec752a" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260807 22:59:40.122839 15466 tablet.cc:2369] T f80dafa100b04f76b5bdf5aae6283da7 P 5c352d1c025747bb86d7f33a5270c5dd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 22:59:40.138134 15766 mvcc.cc:205] Tried to move back new op lower bound from 7316044104145920000 to 7316044103866122240. Current Snapshot: MvccSnapshot[applied={T|T < 7316044104145920000}]
I20260807 22:59:40.161000 14854 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57608:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 22:59:40.161394 14854 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57608:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 22:59:40.166512 14854 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3966b4aeaddc4b73862f071464bbe3d3: Sending DeleteTablet for 3 replicas of tablet f80dafa100b04f76b5bdf5aae6283da7
I20260807 22:59:40.168363 15591 tablet_service.cc:1514] Processing DeleteTablet for tablet f80dafa100b04f76b5bdf5aae6283da7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:59:40 UTC) from {username='kudu', principal='kudu/127.25.94.62@KRBTEST.COM'} at 127.0.0.1:45594
I20260807 22:59:40.170415 15350 tablet_service.cc:1514] Processing DeleteTablet for tablet f80dafa100b04f76b5bdf5aae6283da7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:59:40 UTC) from {username='kudu', principal='kudu/127.25.94.62@KRBTEST.COM'} at 127.0.0.1:52058
I20260807 22:59:40.171283 15112 tablet_service.cc:1514] Processing DeleteTablet for tablet f80dafa100b04f76b5bdf5aae6283da7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:59:40 UTC) from {username='kudu', principal='kudu/127.25.94.62@KRBTEST.COM'} at 127.0.0.1:39262
I20260807 22:59:40.176894 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 14936
I20260807 22:59:40.179979 15780 tablet_replica.cc:330] T f80dafa100b04f76b5bdf5aae6283da7 P 595157aa1d844fe7b30a673e79ec752a: stopping tablet replica
I20260807 22:59:40.186124 15780 raft_consensus.cc:2229] T f80dafa100b04f76b5bdf5aae6283da7 P 595157aa1d844fe7b30a673e79ec752a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 22:59:40.189038 15780 raft_consensus.cc:2258] T f80dafa100b04f76b5bdf5aae6283da7 P 595157aa1d844fe7b30a673e79ec752a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 22:59:40.191933 15780 ts_tablet_manager.cc:1905] T f80dafa100b04f76b5bdf5aae6283da7 P 595157aa1d844fe7b30a673e79ec752a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 22:59:40.194110 15777 tablet_replica.cc:330] T f80dafa100b04f76b5bdf5aae6283da7 P 887cc7d8436e4c79bb7f1d36c89bc491: stopping tablet replica
I20260807 22:59:40.194686 15777 raft_consensus.cc:2229] T f80dafa100b04f76b5bdf5aae6283da7 P 887cc7d8436e4c79bb7f1d36c89bc491 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 22:59:40.195180 15777 pending_rounds.cc:62] T f80dafa100b04f76b5bdf5aae6283da7 P 887cc7d8436e4c79bb7f1d36c89bc491: Trying to abort 2 pending ops.
I20260807 22:59:40.195389 15777 pending_rounds.cc:68] T f80dafa100b04f76b5bdf5aae6283da7 P 887cc7d8436e4c79bb7f1d36c89bc491: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7316044104145920000 op_type: WRITE_OP write_request { tablet_id: "f80dafa100b04f76b5bdf5aae6283da7" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "val" type: INT32 is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\003\00090\000\0001\324\000\000" indirect_data: "<redacted>""" } external_consistency_mode: CLIENT_PROPAGATED authz_token { token_data: "\010\210\311\331\323\006\"8\n\ntest-admin\022*\n 5f5cd2d339d24c8e84e3a1db6d8fb193\020\001\030\001 \001(\001" signature: "<redacted>""D\204h_\237\374\031\366\241R\250\324\342\242\2742-\232\235\"\330s\307\362\017bx\003*\302\021\231\216\205\014\036\224\332\022\352\212\363>\377\304\023\343Y\334E\237\353V\223\351\320\325\032\034\211X\202v\250" signing_key_seq_num: 0 } } request_id { client_id: "b26cb513a35b45cf9f432de63157715e" seq_no: 0 first_incomplete_seq_no: 0 attempt_no: 0 }
W20260807 22:59:40.195978 15281 consensus_peers.cc:487] T f80dafa100b04f76b5bdf5aae6283da7 P 5c352d1c025747bb86d7f33a5270c5dd -> Peer 887cc7d8436e4c79bb7f1d36c89bc491 (127.25.94.3:43173): Couldn't send request to peer 887cc7d8436e4c79bb7f1d36c89bc491. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: STOPPING. This is attempt 1: this message will repeat every 5th retry.
I20260807 22:59:40.196265 15778 tablet_replica.cc:330] T f80dafa100b04f76b5bdf5aae6283da7 P 5c352d1c025747bb86d7f33a5270c5dd: stopping tablet replica
I20260807 22:59:40.197103 15777 pending_rounds.cc:68] T f80dafa100b04f76b5bdf5aae6283da7 P 887cc7d8436e4c79bb7f1d36c89bc491: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7316044103866122240 op_type: NO_OP noop_request { }
I20260807 22:59:40.197852 15778 raft_consensus.cc:2229] T f80dafa100b04f76b5bdf5aae6283da7 P 5c352d1c025747bb86d7f33a5270c5dd [term 1 LEADER]: Raft consensus shutting down.
I20260807 22:59:40.198675 15778 raft_consensus.cc:2258] T f80dafa100b04f76b5bdf5aae6283da7 P 5c352d1c025747bb86d7f33a5270c5dd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 22:59:40.197347 15777 raft_consensus.cc:2866] T f80dafa100b04f76b5bdf5aae6283da7 P 887cc7d8436e4c79bb7f1d36c89bc491 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260807 22:59:40.201361 15778 ts_tablet_manager.cc:1905] T f80dafa100b04f76b5bdf5aae6283da7 P 5c352d1c025747bb86d7f33a5270c5dd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 22:59:40.205873 15777 raft_consensus.cc:2258] T f80dafa100b04f76b5bdf5aae6283da7 P 887cc7d8436e4c79bb7f1d36c89bc491 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 22:59:40.210902 15778 ts_tablet_manager.cc:1918] T f80dafa100b04f76b5bdf5aae6283da7 P 5c352d1c025747bb86d7f33a5270c5dd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 22:59:40.211144 15778 log.cc:1196] T f80dafa100b04f76b5bdf5aae6283da7 P 5c352d1c025747bb86d7f33a5270c5dd: Deleting WAL directory at /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/ts-1/wal/wals/f80dafa100b04f76b5bdf5aae6283da7
I20260807 22:59:40.211796 15778 ts_tablet_manager.cc:1939] T f80dafa100b04f76b5bdf5aae6283da7 P 5c352d1c025747bb86d7f33a5270c5dd: Deleting consensus metadata
I20260807 22:59:40.212677 15780 ts_tablet_manager.cc:1918] T f80dafa100b04f76b5bdf5aae6283da7 P 595157aa1d844fe7b30a673e79ec752a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 22:59:40.212919 15780 log.cc:1196] T f80dafa100b04f76b5bdf5aae6283da7 P 595157aa1d844fe7b30a673e79ec752a: Deleting WAL directory at /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/ts-0/wal/wals/f80dafa100b04f76b5bdf5aae6283da7
I20260807 22:59:40.214955 15780 ts_tablet_manager.cc:1939] T f80dafa100b04f76b5bdf5aae6283da7 P 595157aa1d844fe7b30a673e79ec752a: Deleting consensus metadata
I20260807 22:59:40.215137 15777 ts_tablet_manager.cc:1905] T f80dafa100b04f76b5bdf5aae6283da7 P 887cc7d8436e4c79bb7f1d36c89bc491: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 22:59:40.216816 14822 catalog_manager.cc:4841] TS 5c352d1c025747bb86d7f33a5270c5dd (127.25.94.2:41899): tablet f80dafa100b04f76b5bdf5aae6283da7 (table test-table [id=5f5cd2d339d24c8e84e3a1db6d8fb193]) successfully deleted
I20260807 22:59:40.215026 15727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:40.178505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39274 (local address 127.25.94.1:44253)
0807 22:59:40.178708 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0807 22:59:40.178715 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 22:59:40.188081 (+ 9366us) server_negotiation.cc:407] Connection header received
0807 22:59:40.188342 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:59:40.188351 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:59:40.188498 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:59:40.188615 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:59:40.196137 (+ 7522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:59:40.197318 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:59:40.199598 (+ 2280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:59:40.200236 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:59:40.200350 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:59:40.204350 (+ 4000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:59:40.204365 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:59:40.204388 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:59:40.204432 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:59:40.209159 (+ 4727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:59:40.211075 (+ 1916us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:59:40.211088 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:59:40.211095 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:59:40.211385 (+ 290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:59:40.212179 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:59:40.212185 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:59:40.212189 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:59:40.212766 (+ 577us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:59:40.213394 (+ 628us) server_negotiation.cc:1035] Waiting for connection context
0807 22:59:40.214178 (+ 784us) server_negotiation.cc:299] Negotiation successful
0807 22:59:40.214589 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260807 22:59:40.218114 15195 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 22:59:40.224814 15777 ts_tablet_manager.cc:1918] T f80dafa100b04f76b5bdf5aae6283da7 P 887cc7d8436e4c79bb7f1d36c89bc491: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 22:59:40.225132 15777 log.cc:1196] T f80dafa100b04f76b5bdf5aae6283da7 P 887cc7d8436e4c79bb7f1d36c89bc491: Deleting WAL directory at /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786143361779964-25976-0/minicluster-data/ts-2/wal/wals/f80dafa100b04f76b5bdf5aae6283da7
I20260807 22:59:40.226716 15777 ts_tablet_manager.cc:1939] T f80dafa100b04f76b5bdf5aae6283da7 P 887cc7d8436e4c79bb7f1d36c89bc491: Deleting consensus metadata
I20260807 22:59:40.230726 14822 catalog_manager.cc:4841] TS 887cc7d8436e4c79bb7f1d36c89bc491 (127.25.94.3:43173): tablet f80dafa100b04f76b5bdf5aae6283da7 (table test-table [id=5f5cd2d339d24c8e84e3a1db6d8fb193]) successfully deleted
I20260807 22:59:40.325548 14821 catalog_manager.cc:4841] TS 595157aa1d844fe7b30a673e79ec752a (127.25.94.1:44253): tablet f80dafa100b04f76b5bdf5aae6283da7 (table test-table [id=5f5cd2d339d24c8e84e3a1db6d8fb193]) successfully deleted
I20260807 22:59:40.325951 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 14936
I20260807 22:59:40.354871 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 15216
I20260807 22:59:40.374497 15732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:40.355736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52076 (local address 127.25.94.2:41899)
0807 22:59:40.356023 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0807 22:59:40.356030 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 22:59:40.356054 (+ 24us) server_negotiation.cc:407] Connection header received
0807 22:59:40.356265 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:59:40.356273 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:59:40.356368 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:59:40.356493 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:59:40.358422 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:59:40.359727 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:59:40.361767 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:59:40.362487 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:59:40.362670 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:59:40.366495 (+ 3825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:59:40.366511 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:59:40.366531 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:59:40.366585 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:59:40.371026 (+ 4441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:59:40.372004 (+ 978us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:59:40.372014 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:59:40.372019 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:59:40.372196 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:59:40.372661 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:59:40.372666 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:59:40.372670 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:59:40.373143 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:59:40.373504 (+ 361us) server_negotiation.cc:1035] Waiting for connection context
0807 22:59:40.373815 (+ 311us) server_negotiation.cc:299] Negotiation successful
0807 22:59:40.374049 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144}
I20260807 22:59:40.375289 15457 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 22:59:40.492425 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 15216
I20260807 22:59:40.522886 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 15473
I20260807 22:59:40.537333 15721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:40.523682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45608 (local address 127.25.94.3:43173)
0807 22:59:40.523886 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0807 22:59:40.523893 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 22:59:40.523958 (+ 65us) server_negotiation.cc:407] Connection header received
0807 22:59:40.524180 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:59:40.524186 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:59:40.524272 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:59:40.524390 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:59:40.525853 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:59:40.526572 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:59:40.527858 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:59:40.528255 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:59:40.528397 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:59:40.531296 (+ 2899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:59:40.531309 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:59:40.531326 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:59:40.531364 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:59:40.534287 (+ 2923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:59:40.535053 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:59:40.535065 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:59:40.535074 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:59:40.535225 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:59:40.535633 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:59:40.535639 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:59:40.535642 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:59:40.536002 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:59:40.536310 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0807 22:59:40.536645 (+ 335us) server_negotiation.cc:299] Negotiation successful
0807 22:59:40.536875 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260807 22:59:40.538100 15689 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 22:59:40.627789 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 15473
I20260807 22:59:40.655042 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 14788
I20260807 22:59:40.656311 14922 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 22:59:40.772334 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 14788
I20260807 22:59:40.804003 25976 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260807 22:59:41.150031 25976 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260807 22:59:41.150420 25976 mini_ranger.cc:66] Stopping Ranger...
I20260807 22:59:41.496958 25976 mini_ranger.cc:68] Stopped Ranger
2026-08-07 22:59:41.497 UTC [13031] LOG: received smart shutdown request
2026-08-07 22:59:41.500 UTC [13031] LOG: background worker "logical replication launcher" (PID 13045) exited with exit code 1
2026-08-07 22:59:41.501 UTC [13039] LOG: shutting down
2026-08-07 22:59:41.683 UTC [13031] LOG: database system is shut down
2026-08-07T22:59:41Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (56589 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786143361779964-25976-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 22:59:41 dist-test-slave-z8x9 krb5kdc[15800](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 22:59:41 dist-test-slave-z8x9 krb5kdc[15800](info): set up 2 sockets
Aug 07 22:59:41 dist-test-slave-z8x9 krb5kdc[15800](info): commencing operation
krb5kdc: starting...
W20260807 22:59:43.194926 25976 mini_kdc.cc:121] Time spent starting KDC: real 1.334s user 0.000s sys 0.035s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786143361779964-25976-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786143361779964-25976-0/krb5kdc/test-user.keytab.
Aug 07 22:59:43 dist-test-slave-z8x9 krb5kdc[15800](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143583, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T22:59:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T22:59:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.94.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.62.keytab.
Entry for principal kudu/127.25.94.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.62.keytab.
I20260807 22:59:43.495517 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786143361779964-25976-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786143361779964-25976-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786143361779964-25976-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.62:36419
--webserver_interface=127.25.94.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.62.keytab
--principal=kudu/127.25.94.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.20:45241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.94.62:36419,127.25.94.61:36505,127.25.94.60:34111
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786143361779964-25976-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786143361779964-25976-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786143361779964-25976-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:59:43.841992 15824 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:59:43.842298 15824 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:59:43.842389 15824 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:59:43.849550 15824 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 22:59:43.849661 15824 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:59:43.849707 15824 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:59:43.849769 15824 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 22:59:43.849813 15824 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 22:59:43.859589 15824 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:45241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786143361779964-25976-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786143361779964-25976-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.94.62:36419,127.25.94.61:36505,127.25.94.60:34111
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.62.keytab
--principal=kudu/127.25.94.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.62:36419
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.94.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786143361779964-25976-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 22:59:43.860874 15824 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:59:43.862257 15824 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:59:43.873844 15837 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:59:43.874210 15841 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:59:43.874315 15836 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:59:43.874985 15824 server_base.cc:1034] running on GCE node
I20260807 22:59:43.875749 15824 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:59:43.877084 15824 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:59:43.878383 15824 hybrid_clock.cc:648] HybridClock initialized: now 1786143583878315 us; error 62 us; skew 500 ppm
I20260807 22:59:43.878731 15824 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 22:59:43 dist-test-slave-z8x9 krb5kdc[15800](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143583, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:59:43.887648 15824 init.cc:377] Logged in from keytab as kudu/127.25.94.62@KRBTEST.COM (short username kudu)
I20260807 22:59:43.889294 15824 webserver.cc:460] Webserver started at http://127.25.94.62:33877/ using document root <none> and password file <none>
I20260807 22:59:43.890002 15824 fs_manager.cc:359] Metadata directory not provided
I20260807 22:59:43.890151 15824 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:59:43.890473 15824 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:59:43.893579 15824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786143361779964-25976-0/minicluster-data/master-0/data/instance:
uuid: "4ba7dd4fb2b04e91a64aeb796378871b"
format_stamp: "Formatted at 2026-08-07 22:59:43 on dist-test-slave-z8x9"
server_key: "ad9af9892af45792a579e9eb9ced6746"
server_key_iv: "5e4481f5869a8d6f81e7287056fcc6e8"
server_key_version: "encryptionkey@0"
I20260807 22:59:43.894415 15824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786143361779964-25976-0/minicluster-data/master-0/wal/instance:
uuid: "4ba7dd4fb2b04e91a64aeb796378871b"
format_stamp: "Formatted at 2026-08-07 22:59:43 on dist-test-slave-z8x9"
server_key: "ad9af9892af45792a579e9eb9ced6746"
server_key_iv: "5e4481f5869a8d6f81e7287056fcc6e8"
server_key_version: "encryptionkey@0"
I20260807 22:59:43.899479 15824 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260807 22:59:43.902855 15851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:59:43.904065 15824 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260807 22:59:43.904284 15824 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786143361779964-25976-0/minicluster-data/master-0/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786143361779964-25976-0/minicluster-data/master-0/wal
uuid: "4ba7dd4fb2b04e91a64aeb796378871b"
format_stamp: "Formatted at 2026-08-07 22:59:43 on dist-test-slave-z8x9"
server_key: "ad9af9892af45792a579e9eb9ced6746"
server_key_iv: "5e4481f5869a8d6f81e7287056fcc6e8"
server_key_version: "encryptionkey@0"
I20260807 22:59:43.904491 15824 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786143361779964-25976-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786143361779964-25976-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786143361779964-25976-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:59:43.925714 15824 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:59:43.930398 15824 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:59:43.930721 15824 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:59:43.950579 15824 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.62:36419
I20260807 22:59:43.950619 15942 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.62:36419 every 8 connection(s)
I20260807 22:59:43.952076 15824 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
I20260807 22:59:43.953727 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 15824
I20260807 22:59:43.953974 25976 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786143361779964-25976-0/minicluster-data/master-0/wal/instance
I20260807 22:59:43.954833 25976 external_mini_cluster.cc:1428] Setting key 87b0d3a300de7db88f53c3c1b6c74d6c
I20260807 22:59:43.957056 15948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:59:43.967396 15948 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.94.62" port: 36419 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.25.94.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.61@KRBTEST.COM" created.
Aug 07 22:59:43 dist-test-slave-z8x9 krb5kdc[15800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143583, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.62@KRBTEST.COM for kudu/127.25.94.62@KRBTEST.COM
I20260807 22:59:43.994820 15951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:43.970721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:36419 (local address 127.0.0.1:40478)
0807 22:59:43.971662 (+ 941us) negotiation.cc:107] Waiting for socket to connect
0807 22:59:43.971709 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0807 22:59:43.973810 (+ 2101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:59:43.974660 (+ 850us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:59:43.974666 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:59:43.975415 (+ 749us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:59:43.975760 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:59:43.975779 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:59:43.977486 (+ 1707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:59:43.977493 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:59:43.978569 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:59:43.978581 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:59:43.979542 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:59:43.979549 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:59:43.979717 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:59:43.980466 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:59:43.980488 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:59:43.985649 (+ 5161us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:59:43.990594 (+ 4945us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:59:43.990605 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:59:43.990616 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:59:43.991045 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:59:43.991545 (+ 500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:59:43.991548 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:59:43.991550 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:59:43.991702 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:59:43.992773 (+ 1071us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:59:43.992779 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:59:43.992981 (+ 202us) client_negotiation.cc:769] Sending connection context
0807 22:59:43.993286 (+ 305us) client_negotiation.cc:240] Negotiation successful
0807 22:59:43.993700 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":674,"thread_start_us":420,"threads_started":1}
I20260807 22:59:43.994855 15952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:43.970726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40478 (local address 127.25.94.62:36419)
0807 22:59:43.972499 (+ 1773us) server_negotiation.cc:206] Beginning negotiation
0807 22:59:43.972508 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 22:59:43.972556 (+ 48us) server_negotiation.cc:407] Connection header received
0807 22:59:43.974178 (+ 1622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:59:43.974183 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:59:43.974409 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:59:43.974647 (+ 238us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:59:43.975971 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:59:43.977334 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:59:43.978718 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:59:43.979281 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:59:43.979515 (+ 234us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:59:43.985943 (+ 6428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:59:43.985958 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:59:43.985981 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:59:43.986035 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:59:43.990405 (+ 4370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:59:43.991198 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:59:43.991211 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:59:43.991217 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:59:43.991428 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:59:43.991827 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:59:43.991834 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:59:43.991840 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:59:43.992612 (+ 772us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:59:43.992837 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0807 22:59:43.993452 (+ 615us) server_negotiation.cc:299] Negotiation successful
0807 22:59:43.993923 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":116,"server-negotiator.queue_time_us":1552,"spinlock_wait_cycles":19328,"thread_start_us":1322,"threads_started":1}
I20260807 22:59:43.998306 15948 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.94.61" port: 36505 } has no permanent_uuid. Determining permanent_uuid...
I20260807 22:59:43.999621 15951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:43.999179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.61:36505 (local address 127.0.0.1:37884)
0807 22:59:43.999378 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0807 22:59:43.999499 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.61:36505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260807 22:59:43.999996 15856 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.94.61:36505: connect: Connection refused (error 111)
I20260807 22:59:44.001706 15951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:44.001361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.61:36505 (local address 127.0.0.1:37892)
0807 22:59:44.001494 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0807 22:59:44.001572 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.61:36505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260807 22:59:44.002024 15948 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.94.61:36505: Network error: Client connection negotiation failed: client connection to 127.25.94.61:36505: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.61.keytab.
Entry for principal kudu/127.25.94.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.61.keytab.
I20260807 22:59:44.021266 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786143361779964-25976-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786143361779964-25976-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786143361779964-25976-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786143361779964-25976-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.61:36505
--webserver_interface=127.25.94.61
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.61.keytab
--principal=kudu/127.25.94.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.20:45241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.94.62:36419,127.25.94.61:36505,127.25.94.60:34111
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786143361779964-25976-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786143361779964-25976-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786143361779964-25976-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 22:59:44.051455 15948 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.94.61" port: 36505 } attempt: 1
I20260807 22:59:44.052776 15951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:44.052196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.61:36505 (local address 127.0.0.1:37894)
0807 22:59:44.052468 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0807 22:59:44.052587 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.61:36505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260807 22:59:44.053947 15951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:44.053507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.61:36505 (local address 127.0.0.1:37902)
0807 22:59:44.053659 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0807 22:59:44.053746 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.61:36505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260807 22:59:44.054164 15948 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.94.61:36505: Network error: Client connection negotiation failed: client connection to 127.25.94.61:36505: connect: Connection refused (error 111)
I20260807 22:59:44.122473 15948 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.94.61" port: 36505 } attempt: 2
I20260807 22:59:44.123744 15951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:44.123251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.61:36505 (local address 127.0.0.1:37918)
0807 22:59:44.123483 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0807 22:59:44.123579 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.61:36505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260807 22:59:44.125054 15951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:44.124591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.61:36505 (local address 127.0.0.1:37924)
0807 22:59:44.124803 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0807 22:59:44.124903 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.61:36505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
W20260807 22:59:44.125365 15948 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.94.61:36505: Network error: Client connection negotiation failed: client connection to 127.25.94.61:36505: connect: Connection refused (error 111)
I20260807 22:59:44.216682 15948 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.94.61" port: 36505 } attempt: 3
I20260807 22:59:44.218050 15951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:44.217586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.61:36505 (local address 127.0.0.1:37932)
0807 22:59:44.217801 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0807 22:59:44.217902 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.61:36505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260807 22:59:44.219530 15951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:44.218977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.61:36505 (local address 127.0.0.1:37938)
0807 22:59:44.219177 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0807 22:59:44.219376 (+ 199us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.61:36505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
W20260807 22:59:44.219919 15948 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.94.61:36505: Network error: Client connection negotiation failed: client connection to 127.25.94.61:36505: connect: Connection refused (error 111)
I20260807 22:59:44.363330 15948 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.94.61" port: 36505 } attempt: 4
I20260807 22:59:44.364728 15951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:44.364125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.61:36505 (local address 127.0.0.1:37950)
0807 22:59:44.364513 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0807 22:59:44.364597 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.61:36505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":247}
I20260807 22:59:44.365861 15951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:44.365516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.61:36505 (local address 127.0.0.1:37954)
0807 22:59:44.365652 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0807 22:59:44.365740 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.61:36505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260807 22:59:44.366107 15948 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.94.61:36505: Network error: Client connection negotiation failed: client connection to 127.25.94.61:36505: connect: Connection refused (error 111)
W20260807 22:59:44.379247 15958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:59:44.379565 15958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:59:44.379763 15958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:59:44.388092 15958 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 22:59:44.388234 15958 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:59:44.388317 15958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:59:44.388434 15958 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 22:59:44.388501 15958 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 22:59:44.398864 15958 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:45241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786143361779964-25976-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786143361779964-25976-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.25.94.62:36419,127.25.94.61:36505,127.25.94.60:34111
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.61.keytab
--principal=kudu/127.25.94.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.61:36505
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786143361779964-25976-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.25.94.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786143361779964-25976-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 22:59:44.400293 15958 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:59:44.401664 15958 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:59:44.413894 15966 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:59:44.414093 15969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:59:44.413872 15971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:59:44.415043 15958 server_base.cc:1034] running on GCE node
I20260807 22:59:44.415825 15958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:59:44.417237 15958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:59:44.418493 15958 hybrid_clock.cc:648] HybridClock initialized: now 1786143584418428 us; error 52 us; skew 500 ppm
I20260807 22:59:44.418960 15958 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 22:59:44 dist-test-slave-z8x9 krb5kdc[15800](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143584, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:59:44.422513 15958 init.cc:377] Logged in from keytab as kudu/127.25.94.61@KRBTEST.COM (short username kudu)
I20260807 22:59:44.424029 15958 webserver.cc:460] Webserver started at http://127.25.94.61:40745/ using document root <none> and password file <none>
I20260807 22:59:44.424768 15958 fs_manager.cc:359] Metadata directory not provided
I20260807 22:59:44.424904 15958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:59:44.425302 15958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:59:44.428313 15958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786143361779964-25976-0/minicluster-data/master-1/data/instance:
uuid: "f0cedc141ce141f5b930a1044ca76efa"
format_stamp: "Formatted at 2026-08-07 22:59:44 on dist-test-slave-z8x9"
server_key: "a7eadb775a4d4f03a3f62b581bbc3ca2"
server_key_iv: "f4b98d01d3f6d9259e25cd2ad8140382"
server_key_version: "encryptionkey@0"
I20260807 22:59:44.429217 15958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786143361779964-25976-0/minicluster-data/master-1/wal/instance:
uuid: "f0cedc141ce141f5b930a1044ca76efa"
format_stamp: "Formatted at 2026-08-07 22:59:44 on dist-test-slave-z8x9"
server_key: "a7eadb775a4d4f03a3f62b581bbc3ca2"
server_key_iv: "f4b98d01d3f6d9259e25cd2ad8140382"
server_key_version: "encryptionkey@0"
I20260807 22:59:44.434497 15958 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260807 22:59:44.438036 15980 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:59:44.439457 15958 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 22:59:44.439640 15958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786143361779964-25976-0/minicluster-data/master-1/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786143361779964-25976-0/minicluster-data/master-1/wal
uuid: "f0cedc141ce141f5b930a1044ca76efa"
format_stamp: "Formatted at 2026-08-07 22:59:44 on dist-test-slave-z8x9"
server_key: "a7eadb775a4d4f03a3f62b581bbc3ca2"
server_key_iv: "f4b98d01d3f6d9259e25cd2ad8140382"
server_key_version: "encryptionkey@0"
I20260807 22:59:44.439864 15958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786143361779964-25976-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786143361779964-25976-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786143361779964-25976-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:59:44.475272 15958 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:59:44.479776 15958 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:59:44.480110 15958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:59:44.499974 15958 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.61:36505
I20260807 22:59:44.500010 16075 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.61:36505 every 8 connection(s)
I20260807 22:59:44.501555 15958 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786143361779964-25976-0/minicluster-data/master-1/data/info.pb
I20260807 22:59:44.505693 16076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:59:44.511106 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 15958
I20260807 22:59:44.511336 25976 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786143361779964-25976-0/minicluster-data/master-1/wal/instance
I20260807 22:59:44.511895 25976 external_mini_cluster.cc:1428] Setting key 8dc0f15d7067652989dc017231961688
I20260807 22:59:44.513518 16076 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.94.62" port: 36419 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 22:59:44 dist-test-slave-z8x9 krb5kdc[15800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143584, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.61@KRBTEST.COM for kudu/127.25.94.62@KRBTEST.COM
I20260807 22:59:44.539453 16078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:44.517470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40480 (local address 127.25.94.62:36419)
0807 22:59:44.518067 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0807 22:59:44.518074 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 22:59:44.519184 (+ 1110us) server_negotiation.cc:407] Connection header received
0807 22:59:44.522209 (+ 3025us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:59:44.522213 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:59:44.522330 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:59:44.522437 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:59:44.524501 (+ 2064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:59:44.525346 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:59:44.526942 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:59:44.527353 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:59:44.527490 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:59:44.531428 (+ 3938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:59:44.531439 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:59:44.531447 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:59:44.531513 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:59:44.535598 (+ 4085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:59:44.536507 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:59:44.536515 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:59:44.536521 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:59:44.536691 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:59:44.537220 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:59:44.537229 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:59:44.537235 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:59:44.537710 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:59:44.537895 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0807 22:59:44.538664 (+ 769us) server_negotiation.cc:299] Negotiation successful
0807 22:59:44.538934 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":451,"thread_start_us":293,"threads_started":1}
I20260807 22:59:44.540148 16080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:44.518060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:36419 (local address 127.0.0.1:40480)
0807 22:59:44.518956 (+ 896us) negotiation.cc:107] Waiting for socket to connect
0807 22:59:44.519017 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0807 22:59:44.521867 (+ 2850us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:59:44.522805 (+ 938us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:59:44.522820 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:59:44.523645 (+ 825us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:59:44.524288 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:59:44.524307 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:59:44.525519 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:59:44.525523 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:59:44.526773 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:59:44.526785 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:59:44.527589 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:59:44.527597 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:59:44.527825 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:59:44.528552 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:59:44.528575 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:59:44.531126 (+ 2551us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:59:44.535805 (+ 4679us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:59:44.535815 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:59:44.535830 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:59:44.536355 (+ 525us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:59:44.536834 (+ 479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:59:44.536838 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:59:44.536840 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:59:44.537027 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:59:44.537867 (+ 840us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:59:44.537874 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:59:44.538245 (+ 371us) client_negotiation.cc:769] Sending connection context
0807 22:59:44.538503 (+ 258us) client_negotiation.cc:240] Negotiation successful
0807 22:59:44.538945 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":331,"threads_started":1}
WARNING: no policy specified for kudu/127.25.94.60@KRBTEST.COM; defaulting to no policy
I20260807 22:59:44.542114 16076 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.94.61" port: 36505 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.60@KRBTEST.COM" created.
Aug 07 22:59:44 dist-test-slave-z8x9 krb5kdc[15800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143584, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.61@KRBTEST.COM for kudu/127.25.94.61@KRBTEST.COM
I20260807 22:59:44.559079 16080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:44.542957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.61:36505 (local address 127.0.0.1:37958)
0807 22:59:44.543251 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0807 22:59:44.543269 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 22:59:44.543366 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:59:44.544666 (+ 1300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:59:44.544672 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:59:44.545119 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:59:44.545358 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:59:44.545367 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:59:44.547434 (+ 2067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:59:44.547438 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:59:44.548276 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:59:44.548287 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:59:44.549594 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:59:44.549598 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:59:44.549747 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:59:44.550408 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:59:44.550425 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:59:44.552454 (+ 2029us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:59:44.556066 (+ 3612us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:59:44.556075 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:59:44.556082 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:59:44.556612 (+ 530us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:59:44.556979 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:59:44.556986 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:59:44.556992 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:59:44.557220 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:59:44.557843 (+ 623us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:59:44.557851 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:59:44.558051 (+ 200us) client_negotiation.cc:769] Sending connection context
0807 22:59:44.558218 (+ 167us) client_negotiation.cc:240] Negotiation successful
0807 22:59:44.558553 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146}
I20260807 22:59:44.559080 16083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:44.543081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37958 (local address 127.25.94.61:36505)
0807 22:59:44.543957 (+ 876us) server_negotiation.cc:206] Beginning negotiation
0807 22:59:44.543963 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 22:59:44.543989 (+ 26us) server_negotiation.cc:407] Connection header received
0807 22:59:44.544033 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:59:44.544039 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:59:44.544286 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:59:44.544490 (+ 204us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:59:44.545536 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:59:44.547277 (+ 1741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:59:44.548423 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:59:44.549452 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:59:44.549601 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:59:44.552680 (+ 3079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:59:44.552697 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:59:44.552718 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:59:44.552771 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:59:44.555929 (+ 3158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:59:44.556738 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:59:44.556744 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:59:44.556747 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:59:44.556872 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:59:44.557349 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:59:44.557352 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:59:44.557354 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:59:44.557698 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:59:44.557867 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0807 22:59:44.558349 (+ 482us) server_negotiation.cc:299] Negotiation successful
0807 22:59:44.558669 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":115,"server-negotiator.queue_time_us":677,"thread_start_us":518,"threads_started":1}
I20260807 22:59:44.561597 16076 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.94.60" port: 34111 } has no permanent_uuid. Determining permanent_uuid...
I20260807 22:59:44.562884 16080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:44.562406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.60:34111 (local address 127.0.0.1:34026)
0807 22:59:44.562593 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0807 22:59:44.562711 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.60:34111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260807 22:59:44.563302 15984 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.94.60:34111: connect: Connection refused (error 111)
I20260807 22:59:44.564898 16080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:44.564588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.60:34111 (local address 127.0.0.1:34042)
0807 22:59:44.564721 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0807 22:59:44.564802 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.60:34111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260807 22:59:44.565279 16076 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.94.60:34111: Network error: Client connection negotiation failed: client connection to 127.25.94.60:34111: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.60.keytab.
Entry for principal kudu/127.25.94.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.60.keytab.
I20260807 22:59:44.581501 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786143361779964-25976-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786143361779964-25976-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786143361779964-25976-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786143361779964-25976-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.60:34111
--webserver_interface=127.25.94.60
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.60.keytab
--principal=kudu/127.25.94.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.20:45241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.94.62:36419,127.25.94.61:36505,127.25.94.60:34111
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786143361779964-25976-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786143361779964-25976-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786143361779964-25976-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 22:59:44.614598 16076 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.94.60" port: 34111 } attempt: 1
I20260807 22:59:44.615895 16080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:44.615374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.60:34111 (local address 127.0.0.1:34048)
0807 22:59:44.615616 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0807 22:59:44.615736 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.60:34111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260807 22:59:44.616968 16080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:44.616625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.60:34111 (local address 127.0.0.1:34062)
0807 22:59:44.616780 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0807 22:59:44.616862 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.60:34111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260807 22:59:44.617218 16076 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.94.60:34111: Network error: Client connection negotiation failed: client connection to 127.25.94.60:34111: connect: Connection refused (error 111)
I20260807 22:59:44.665383 15948 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.94.61" port: 36505 } attempt: 5
Aug 07 22:59:44 dist-test-slave-z8x9 krb5kdc[15800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143583, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.62@KRBTEST.COM for kudu/127.25.94.61@KRBTEST.COM
I20260807 22:59:44.682075 15951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:44.666537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.61:36505 (local address 127.0.0.1:37968)
0807 22:59:44.667736 (+ 1199us) negotiation.cc:107] Waiting for socket to connect
0807 22:59:44.667771 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0807 22:59:44.667939 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:59:44.668451 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:59:44.668457 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:59:44.668945 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:59:44.669121 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:59:44.669131 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:59:44.670630 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:59:44.670634 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:59:44.671667 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:59:44.671681 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:59:44.672671 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:59:44.672676 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:59:44.672807 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:59:44.673511 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:59:44.673530 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:59:44.675808 (+ 2278us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:59:44.679723 (+ 3915us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:59:44.679727 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:59:44.679729 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:59:44.680106 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:59:44.680508 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:59:44.680512 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:59:44.680514 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:59:44.680638 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:59:44.681267 (+ 629us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:59:44.681270 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:59:44.681401 (+ 131us) client_negotiation.cc:769] Sending connection context
0807 22:59:44.681539 (+ 138us) client_negotiation.cc:240] Negotiation successful
0807 22:59:44.681820 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1047}
I20260807 22:59:44.682224 16083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:44.666720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37968 (local address 127.25.94.61:36505)
0807 22:59:44.666932 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0807 22:59:44.666939 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 22:59:44.668084 (+ 1145us) server_negotiation.cc:407] Connection header received
0807 22:59:44.668163 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:59:44.668169 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:59:44.668276 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:59:44.668391 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:59:44.669347 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:59:44.670440 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:59:44.671826 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:59:44.672503 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:59:44.672633 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:59:44.676042 (+ 3409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:59:44.676051 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:59:44.676058 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:59:44.676103 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:59:44.679457 (+ 3354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:59:44.680239 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:59:44.680245 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:59:44.680248 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:59:44.680397 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:59:44.680741 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:59:44.680746 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:59:44.680749 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:59:44.681104 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:59:44.681320 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0807 22:59:44.681688 (+ 368us) server_negotiation.cc:299] Negotiation successful
0807 22:59:44.681915 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260807 22:59:44.683302 15948 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.94.60" port: 34111 } has no permanent_uuid. Determining permanent_uuid...
I20260807 22:59:44.684401 15951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:44.684027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.60:34111 (local address 127.0.0.1:34072)
0807 22:59:44.684187 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0807 22:59:44.684260 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.60:34111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260807 22:59:44.685348 15951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:44.684967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.60:34111 (local address 127.0.0.1:34080)
0807 22:59:44.685104 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0807 22:59:44.685228 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.60:34111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260807 22:59:44.685427 16076 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.94.60" port: 34111 } attempt: 2
W20260807 22:59:44.685817 15948 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.94.60:34111: Network error: Client connection negotiation failed: client connection to 127.25.94.60:34111: connect: Connection refused (error 111)
I20260807 22:59:44.686281 16080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:44.685904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.60:34111 (local address 127.0.0.1:34090)
0807 22:59:44.686083 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0807 22:59:44.686163 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.60:34111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260807 22:59:44.687215 16080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:44.686894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.60:34111 (local address 127.0.0.1:34106)
0807 22:59:44.687037 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0807 22:59:44.687110 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.60:34111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260807 22:59:44.687430 16076 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.94.60:34111: Network error: Client connection negotiation failed: client connection to 127.25.94.60:34111: connect: Connection refused (error 111)
I20260807 22:59:44.737104 15948 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.94.60" port: 34111 } attempt: 1
I20260807 22:59:44.738329 15951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:44.737867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.60:34111 (local address 127.0.0.1:34112)
0807 22:59:44.738067 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0807 22:59:44.738161 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.60:34111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260807 22:59:44.739283 15951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:44.738957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.60:34111 (local address 127.0.0.1:34122)
0807 22:59:44.739111 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0807 22:59:44.739183 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.60:34111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260807 22:59:44.739507 15948 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.94.60:34111: Network error: Client connection negotiation failed: client connection to 127.25.94.60:34111: connect: Connection refused (error 111)
I20260807 22:59:44.778663 16076 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.94.60" port: 34111 } attempt: 3
I20260807 22:59:44.779968 16080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:44.779477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.60:34111 (local address 127.0.0.1:34124)
0807 22:59:44.779703 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0807 22:59:44.779807 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.60:34111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260807 22:59:44.781003 16080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:44.780607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.60:34111 (local address 127.0.0.1:34138)
0807 22:59:44.780832 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0807 22:59:44.780904 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.60:34111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
W20260807 22:59:44.781239 16076 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.94.60:34111: Network error: Client connection negotiation failed: client connection to 127.25.94.60:34111: connect: Connection refused (error 111)
I20260807 22:59:44.807725 15948 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.94.60" port: 34111 } attempt: 2
I20260807 22:59:44.809000 15951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:44.808448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.60:34111 (local address 127.0.0.1:34146)
0807 22:59:44.808714 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0807 22:59:44.808842 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.60:34111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260807 22:59:44.810119 15951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:44.809699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.60:34111 (local address 127.0.0.1:34160)
0807 22:59:44.809861 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0807 22:59:44.809954 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.60:34111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260807 22:59:44.810549 15948 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.94.60:34111: Network error: Client connection negotiation failed: client connection to 127.25.94.60:34111: connect: Connection refused (error 111)
I20260807 22:59:44.916774 15948 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.94.60" port: 34111 } attempt: 3
I20260807 22:59:44.918094 15951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:44.917522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.60:34111 (local address 127.0.0.1:34166)
0807 22:59:44.917775 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0807 22:59:44.917888 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.60:34111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260807 22:59:44.919180 15951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:44.918734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.60:34111 (local address 127.0.0.1:34170)
0807 22:59:44.918951 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0807 22:59:44.919042 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.60:34111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
W20260807 22:59:44.919421 15948 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.94.60:34111: Network error: Client connection negotiation failed: client connection to 127.25.94.60:34111: connect: Connection refused (error 111)
I20260807 22:59:44.924496 16076 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.94.60" port: 34111 } attempt: 4
I20260807 22:59:44.925606 16080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:44.925073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.60:34111 (local address 127.0.0.1:34176)
0807 22:59:44.925330 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0807 22:59:44.925430 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.60:34111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260807 22:59:44.926645 16080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:44.926284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.60:34111 (local address 127.0.0.1:34186)
0807 22:59:44.926436 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0807 22:59:44.926522 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.60:34111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260807 22:59:44.926877 16076 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.94.60:34111: Network error: Client connection negotiation failed: client connection to 127.25.94.60:34111: connect: Connection refused (error 111)
W20260807 22:59:44.931474 16088 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:59:44.931774 16088 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:59:44.931915 16088 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:59:44.939447 16088 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 22:59:44.939558 16088 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:59:44.939659 16088 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:59:44.939757 16088 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 22:59:44.939831 16088 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 22:59:44.950184 16088 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:45241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786143361779964-25976-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786143361779964-25976-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.25.94.62:36419,127.25.94.61:36505,127.25.94.60:34111
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.60.keytab
--principal=kudu/127.25.94.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.60:34111
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786143361779964-25976-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.25.94.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786143361779964-25976-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 22:59:44.951534 16088 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:59:44.952850 16088 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:59:44.965026 16100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:59:44.964999 16102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:59:44.965076 16097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:59:44.966323 16088 server_base.cc:1034] running on GCE node
I20260807 22:59:44.967330 16088 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:59:44.968858 16088 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:59:44.970105 16088 hybrid_clock.cc:648] HybridClock initialized: now 1786143584970045 us; error 56 us; skew 500 ppm
I20260807 22:59:44.970507 16088 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 22:59:44 dist-test-slave-z8x9 krb5kdc[15800](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143584, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:59:44.973868 16088 init.cc:377] Logged in from keytab as kudu/127.25.94.60@KRBTEST.COM (short username kudu)
I20260807 22:59:44.975373 16088 webserver.cc:460] Webserver started at http://127.25.94.60:46367/ using document root <none> and password file <none>
I20260807 22:59:44.976080 16088 fs_manager.cc:359] Metadata directory not provided
I20260807 22:59:44.976224 16088 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:59:44.976531 16088 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:59:44.979926 16088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786143361779964-25976-0/minicluster-data/master-2/data/instance:
uuid: "b31b5e7eb77a404a99709ca318ecadf4"
format_stamp: "Formatted at 2026-08-07 22:59:44 on dist-test-slave-z8x9"
server_key: "1ce7cdfe3daaa591c935981f879b2a9f"
server_key_iv: "e60d1a194e51e63ee988a7e387f9ae00"
server_key_version: "encryptionkey@0"
I20260807 22:59:44.980859 16088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786143361779964-25976-0/minicluster-data/master-2/wal/instance:
uuid: "b31b5e7eb77a404a99709ca318ecadf4"
format_stamp: "Formatted at 2026-08-07 22:59:44 on dist-test-slave-z8x9"
server_key: "1ce7cdfe3daaa591c935981f879b2a9f"
server_key_iv: "e60d1a194e51e63ee988a7e387f9ae00"
server_key_version: "encryptionkey@0"
I20260807 22:59:44.986541 16088 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260807 22:59:44.990104 16115 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:59:44.991426 16088 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 22:59:44.991624 16088 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786143361779964-25976-0/minicluster-data/master-2/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786143361779964-25976-0/minicluster-data/master-2/wal
uuid: "b31b5e7eb77a404a99709ca318ecadf4"
format_stamp: "Formatted at 2026-08-07 22:59:44 on dist-test-slave-z8x9"
server_key: "1ce7cdfe3daaa591c935981f879b2a9f"
server_key_iv: "e60d1a194e51e63ee988a7e387f9ae00"
server_key_version: "encryptionkey@0"
I20260807 22:59:44.991842 16088 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786143361779964-25976-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786143361779964-25976-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786143361779964-25976-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:59:45.011307 16088 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:59:45.015589 16088 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:59:45.015838 16088 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:59:45.034551 16088 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.60:34111
I20260807 22:59:45.034590 16203 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.60:34111 every 8 connection(s)
I20260807 22:59:45.036005 16088 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786143361779964-25976-0/minicluster-data/master-2/data/info.pb
I20260807 22:59:45.038432 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 16088
I20260807 22:59:45.038650 25976 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786143361779964-25976-0/minicluster-data/master-2/wal/instance
I20260807 22:59:45.039287 25976 external_mini_cluster.cc:1428] Setting key 36cde7d417808fbbe31fb235adb100b5
I20260807 22:59:45.040606 16204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:59:45.050745 16204 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.94.62" port: 36419 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 22:59:45 dist-test-slave-z8x9 krb5kdc[15800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143583, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.60@KRBTEST.COM
Aug 07 22:59:45 dist-test-slave-z8x9 krb5kdc[15800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143584, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.60@KRBTEST.COM for kudu/127.25.94.62@KRBTEST.COM
I20260807 22:59:45.065110 16208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:45.041392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34194 (local address 127.25.94.60:34111)
0807 22:59:45.042228 (+ 836us) server_negotiation.cc:206] Beginning negotiation
0807 22:59:45.042241 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0807 22:59:45.042285 (+ 44us) server_negotiation.cc:407] Connection header received
0807 22:59:45.044094 (+ 1809us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:59:45.044117 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:59:45.044554 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:59:45.044967 (+ 413us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:59:45.046951 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:59:45.048337 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:59:45.050284 (+ 1947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:59:45.050950 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:59:45.051089 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:59:45.056370 (+ 5281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:59:45.056378 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:59:45.056389 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:59:45.056425 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:59:45.060349 (+ 3924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:59:45.061341 (+ 992us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:59:45.061352 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:59:45.061362 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:59:45.061588 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:59:45.062056 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:59:45.062060 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:59:45.062062 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:59:45.062667 (+ 605us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:59:45.063085 (+ 418us) server_negotiation.cc:1035] Waiting for connection context
0807 22:59:45.063405 (+ 320us) server_negotiation.cc:299] Negotiation successful
0807 22:59:45.063867 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":611,"thread_start_us":394,"threads_started":1}
I20260807 22:59:45.071218 16210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:45.053594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:36419 (local address 127.0.0.1:40486)
0807 22:59:45.054560 (+ 966us) negotiation.cc:107] Waiting for socket to connect
0807 22:59:45.054607 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0807 22:59:45.054764 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:59:45.055324 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:59:45.055332 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:59:45.056249 (+ 917us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:59:45.056616 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:59:45.056627 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:59:45.058233 (+ 1606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:59:45.058240 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:59:45.059453 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:59:45.059477 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:59:45.060563 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:59:45.060571 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:59:45.060786 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:59:45.061546 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:59:45.061612 (+ 66us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:59:45.064410 (+ 2798us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:59:45.067960 (+ 3550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:59:45.067972 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:59:45.067978 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:59:45.068659 (+ 681us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:59:45.069232 (+ 573us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:59:45.069243 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:59:45.069247 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:59:45.069408 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:59:45.070055 (+ 647us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:59:45.070063 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:59:45.070250 (+ 187us) client_negotiation.cc:769] Sending connection context
0807 22:59:45.070436 (+ 186us) client_negotiation.cc:240] Negotiation successful
0807 22:59:45.070776 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":722,"spinlock_wait_cycles":14592,"thread_start_us":568,"threads_started":1}
I20260807 22:59:45.071219 16209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:45.053633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40486 (local address 127.25.94.62:36419)
0807 22:59:45.054193 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0807 22:59:45.054199 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 22:59:45.054711 (+ 512us) server_negotiation.cc:407] Connection header received
0807 22:59:45.054911 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:59:45.054919 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:59:45.055033 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:59:45.055141 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:59:45.056798 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:59:45.057993 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:59:45.059615 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:59:45.060345 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:59:45.060487 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:59:45.064602 (+ 4115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:59:45.064608 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:59:45.064611 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:59:45.064646 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:59:45.067826 (+ 3180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:59:45.068801 (+ 975us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:59:45.068810 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:59:45.068814 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:59:45.069058 (+ 244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:59:45.069533 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:59:45.069539 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:59:45.069542 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:59:45.069901 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:59:45.070066 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0807 22:59:45.070531 (+ 465us) server_negotiation.cc:299] Negotiation successful
0807 22:59:45.070757 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":412,"thread_start_us":252,"threads_started":1}
I20260807 22:59:45.072758 16204 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.94.61" port: 36505 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 22:59:45 dist-test-slave-z8x9 krb5kdc[15800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143583, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.62@KRBTEST.COM
Aug 07 22:59:45 dist-test-slave-z8x9 krb5kdc[15800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143584, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.60@KRBTEST.COM for kudu/127.25.94.61@KRBTEST.COM
I20260807 22:59:45.085646 16212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:45.068595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40496 (local address 127.25.94.62:36419)
0807 22:59:45.069380 (+ 785us) server_negotiation.cc:206] Beginning negotiation
0807 22:59:45.069384 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 22:59:45.069404 (+ 20us) server_negotiation.cc:407] Connection header received
0807 22:59:45.069443 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:59:45.069446 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:59:45.069508 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:59:45.069629 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:59:45.071176 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:59:45.072386 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:59:45.074316 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:59:45.074892 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:59:45.075020 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:59:45.079640 (+ 4620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:59:45.079648 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:59:45.079653 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:59:45.079694 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:59:45.082680 (+ 2986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:59:45.083435 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:59:45.083441 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:59:45.083445 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:59:45.083575 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:59:45.083964 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:59:45.083969 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:59:45.083973 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:59:45.084344 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:59:45.084633 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0807 22:59:45.084932 (+ 299us) server_negotiation.cc:299] Negotiation successful
0807 22:59:45.085158 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":665,"thread_start_us":331,"threads_started":1}
I20260807 22:59:45.086308 16210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:45.073499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.61:36505 (local address 127.0.0.1:37984)
0807 22:59:45.073654 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0807 22:59:45.073674 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 22:59:45.073759 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:59:45.074016 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:59:45.074019 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:59:45.074373 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:59:45.074514 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:59:45.074522 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:59:45.075669 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:59:45.075674 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:59:45.076541 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:59:45.076553 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:59:45.077324 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:59:45.077329 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:59:45.077541 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:59:45.078080 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:59:45.078102 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:59:45.080454 (+ 2352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:59:45.083732 (+ 3278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:59:45.083737 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:59:45.083741 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:59:45.084207 (+ 466us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:59:45.084589 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:59:45.084594 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:59:45.084598 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:59:45.084767 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:59:45.085343 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:59:45.085348 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:59:45.085508 (+ 160us) client_negotiation.cc:769] Sending connection context
0807 22:59:45.085664 (+ 156us) client_negotiation.cc:240] Negotiation successful
0807 22:59:45.085948 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260807 22:59:45.086360 16083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:45.073597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37984 (local address 127.25.94.61:36505)
0807 22:59:45.073765 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0807 22:59:45.073769 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 22:59:45.073786 (+ 17us) server_negotiation.cc:407] Connection header received
0807 22:59:45.073835 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:59:45.073839 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:59:45.073895 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:59:45.073981 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:59:45.074675 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:59:45.075511 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:59:45.076702 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:59:45.077118 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:59:45.077311 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:59:45.080633 (+ 3322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:59:45.080638 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:59:45.080641 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:59:45.080676 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:59:45.083575 (+ 2899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:59:45.084314 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:59:45.084320 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:59:45.084324 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:59:45.084471 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:59:45.084885 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:59:45.084890 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:59:45.084893 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:59:45.085219 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:59:45.085388 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0807 22:59:45.085753 (+ 365us) server_negotiation.cc:299] Negotiation successful
0807 22:59:45.085973 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260807 22:59:45.087610 16204 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.94.60" port: 34111 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 22:59:45 dist-test-slave-z8x9 krb5kdc[15800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143584, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.60@KRBTEST.COM for kudu/127.25.94.60@KRBTEST.COM
I20260807 22:59:45.096647 15948 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.94.60" port: 34111 } attempt: 4
Aug 07 22:59:45 dist-test-slave-z8x9 krb5kdc[15800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143583, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.61@KRBTEST.COM
I20260807 22:59:45.104424 16210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:45.088512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.60:34111 (local address 127.0.0.1:34200)
0807 22:59:45.088683 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0807 22:59:45.088705 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 22:59:45.088825 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:59:45.089152 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:59:45.089158 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:59:45.089704 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:59:45.089903 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:59:45.089916 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:59:45.091018 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:59:45.091023 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:59:45.092160 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:59:45.092178 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:59:45.092974 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:59:45.092980 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:59:45.093192 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:59:45.093960 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:59:45.093978 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:59:45.096852 (+ 2874us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:59:45.101398 (+ 4546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:59:45.101407 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:59:45.101413 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:59:45.101918 (+ 505us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:59:45.102373 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:59:45.102381 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:59:45.102387 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:59:45.102585 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:59:45.103263 (+ 678us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:59:45.103267 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:59:45.103451 (+ 184us) client_negotiation.cc:769] Sending connection context
0807 22:59:45.103648 (+ 197us) client_negotiation.cc:240] Negotiation successful
0807 22:59:45.103998 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260807 22:59:45.104704 16208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:45.088422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34200 (local address 127.25.94.60:34111)
0807 22:59:45.088612 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0807 22:59:45.088618 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 22:59:45.088761 (+ 143us) server_negotiation.cc:407] Connection header received
0807 22:59:45.088932 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:59:45.088936 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:59:45.089006 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:59:45.089084 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:59:45.090078 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:59:45.090843 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:59:45.092349 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:59:45.092797 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:59:45.092903 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:59:45.097065 (+ 4162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:59:45.097073 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:59:45.097079 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:59:45.097123 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:59:45.101251 (+ 4128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:59:45.102054 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:59:45.102061 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:59:45.102065 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:59:45.102238 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:59:45.102720 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:59:45.102727 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:59:45.102730 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:59:45.103117 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:59:45.103262 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0807 22:59:45.103763 (+ 501us) server_negotiation.cc:299] Negotiation successful
0807 22:59:45.104294 (+ 531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":219,"server-negotiator.queue_time_us":65}
I20260807 22:59:45.107595 16083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:45.087545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38000 (local address 127.25.94.61:36505)
0807 22:59:45.087969 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0807 22:59:45.087973 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 22:59:45.087986 (+ 13us) server_negotiation.cc:407] Connection header received
0807 22:59:45.088048 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:59:45.088053 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:59:45.088130 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:59:45.088241 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:59:45.091001 (+ 2760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:59:45.091690 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:59:45.093393 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:59:45.094052 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:59:45.094152 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:59:45.100367 (+ 6215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:59:45.100378 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:59:45.100385 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:59:45.100428 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:59:45.104099 (+ 3671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:59:45.105123 (+ 1024us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:59:45.105129 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:59:45.105133 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:59:45.105381 (+ 248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:59:45.105849 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:59:45.105856 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:59:45.105860 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:59:45.106295 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:59:45.106552 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0807 22:59:45.106960 (+ 408us) server_negotiation.cc:299] Negotiation successful
0807 22:59:45.107240 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330}
Aug 07 22:59:45 dist-test-slave-z8x9 krb5kdc[15800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143583, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.62@KRBTEST.COM for kudu/127.25.94.60@KRBTEST.COM
I20260807 22:59:45.111199 16204 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b31b5e7eb77a404a99709ca318ecadf4: Bootstrap starting.
I20260807 22:59:45.114135 25976 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 22:59:45.115621 16204 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b31b5e7eb77a404a99709ca318ecadf4: Neither blocks nor log segments found. Creating new log.
I20260807 22:59:45.116799 15951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:45.097663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.60:34111 (local address 127.0.0.1:34212)
0807 22:59:45.098136 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0807 22:59:45.098160 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0807 22:59:45.098287 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:59:45.098747 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:59:45.098754 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:59:45.099191 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:59:45.099352 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:59:45.099361 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:59:45.101081 (+ 1720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:59:45.101088 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:59:45.102316 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:59:45.102335 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:59:45.104885 (+ 2550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:59:45.104892 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:59:45.105113 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:59:45.105962 (+ 849us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:59:45.105990 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:59:45.109001 (+ 3011us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:59:45.113810 (+ 4809us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:59:45.113817 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:59:45.113822 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:59:45.114437 (+ 615us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:59:45.114883 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:59:45.114888 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:59:45.114893 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:59:45.115058 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:59:45.115676 (+ 618us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:59:45.115681 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:59:45.115872 (+ 191us) client_negotiation.cc:769] Sending connection context
0807 22:59:45.116053 (+ 181us) client_negotiation.cc:240] Negotiation successful
0807 22:59:45.116403 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305}
I20260807 22:59:45.117224 16214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:45.097732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34212 (local address 127.25.94.60:34111)
0807 22:59:45.098432 (+ 700us) server_negotiation.cc:206] Beginning negotiation
0807 22:59:45.098437 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 22:59:45.098459 (+ 22us) server_negotiation.cc:407] Connection header received
0807 22:59:45.098519 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:59:45.098522 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:59:45.098573 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:59:45.098707 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:59:45.099533 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:59:45.100908 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:59:45.102486 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:59:45.103107 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:59:45.103244 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:59:45.109324 (+ 6080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:59:45.109338 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:59:45.109342 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:59:45.109376 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:59:45.113647 (+ 4271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:59:45.114577 (+ 930us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:59:45.114584 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:59:45.114588 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:59:45.114750 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:59:45.115182 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:59:45.115187 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:59:45.115191 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:59:45.115522 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:59:45.115848 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0807 22:59:45.116164 (+ 316us) server_negotiation.cc:299] Negotiation successful
0807 22:59:45.116389 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":555,"thread_start_us":620,"threads_started":1}
I20260807 22:59:45.117190 16204 log.cc:824] T 00000000000000000000000000000000 P b31b5e7eb77a404a99709ca318ecadf4: Log is configured to *not* fsync() on all Append() calls
I20260807 22:59:45.122952 15948 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ba7dd4fb2b04e91a64aeb796378871b: Bootstrap starting.
I20260807 22:59:45.123392 16204 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b31b5e7eb77a404a99709ca318ecadf4: No bootstrap required, opened a new log
I20260807 22:59:45.128202 15948 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4ba7dd4fb2b04e91a64aeb796378871b: Neither blocks nor log segments found. Creating new log.
I20260807 22:59:45.129988 15948 log.cc:824] T 00000000000000000000000000000000 P 4ba7dd4fb2b04e91a64aeb796378871b: Log is configured to *not* fsync() on all Append() calls
I20260807 22:59:45.134459 16204 raft_consensus.cc:348] T 00000000000000000000000000000000 P b31b5e7eb77a404a99709ca318ecadf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ba7dd4fb2b04e91a64aeb796378871b" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 36419 } } peers { permanent_uuid: "f0cedc141ce141f5b930a1044ca76efa" member_type: VOTER last_known_addr { host: "127.25.94.61" port: 36505 } } peers { permanent_uuid: "b31b5e7eb77a404a99709ca318ecadf4" member_type: VOTER last_known_addr { host: "127.25.94.60" port: 34111 } }
I20260807 22:59:45.135008 16204 raft_consensus.cc:374] T 00000000000000000000000000000000 P b31b5e7eb77a404a99709ca318ecadf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:59:45.135138 16204 raft_consensus.cc:729] T 00000000000000000000000000000000 P b31b5e7eb77a404a99709ca318ecadf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b31b5e7eb77a404a99709ca318ecadf4, State: Initialized, Role: FOLLOWER
I20260807 22:59:45.136054 16204 consensus_queue.cc:260] T 00000000000000000000000000000000 P b31b5e7eb77a404a99709ca318ecadf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ba7dd4fb2b04e91a64aeb796378871b" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 36419 } } peers { permanent_uuid: "f0cedc141ce141f5b930a1044ca76efa" member_type: VOTER last_known_addr { host: "127.25.94.61" port: 36505 } } peers { permanent_uuid: "b31b5e7eb77a404a99709ca318ecadf4" member_type: VOTER last_known_addr { host: "127.25.94.60" port: 34111 } }
I20260807 22:59:45.145421 16204 sys_catalog.cc:564] T 00000000000000000000000000000000 P b31b5e7eb77a404a99709ca318ecadf4 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 22:59:45.147101 16236 sys_catalog.cc:455] T 00000000000000000000000000000000 P b31b5e7eb77a404a99709ca318ecadf4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ba7dd4fb2b04e91a64aeb796378871b" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 36419 } } peers { permanent_uuid: "f0cedc141ce141f5b930a1044ca76efa" member_type: VOTER last_known_addr { host: "127.25.94.61" port: 36505 } } peers { permanent_uuid: "b31b5e7eb77a404a99709ca318ecadf4" member_type: VOTER last_known_addr { host: "127.25.94.60" port: 34111 } } }
I20260807 22:59:45.147765 16236 sys_catalog.cc:458] T 00000000000000000000000000000000 P b31b5e7eb77a404a99709ca318ecadf4 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 22:59:45.148813 16212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:45.121819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40508 (local address 127.25.94.62:36419)
0807 22:59:45.122040 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0807 22:59:45.122046 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 22:59:45.122067 (+ 21us) server_negotiation.cc:407] Connection header received
0807 22:59:45.125578 (+ 3511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:59:45.125584 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:59:45.125672 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:59:45.125744 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:59:45.128170 (+ 2426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:59:45.129364 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:59:45.131810 (+ 2446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:59:45.132515 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:59:45.132624 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:59:45.137543 (+ 4919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:59:45.137554 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:59:45.137562 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:59:45.137631 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:59:45.142635 (+ 5004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:59:45.144290 (+ 1655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:59:45.144299 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:59:45.144302 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:59:45.144491 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:59:45.145060 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:59:45.145066 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:59:45.145069 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:59:45.145497 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:59:45.145850 (+ 353us) server_negotiation.cc:1035] Waiting for connection context
0807 22:59:45.147443 (+ 1593us) server_negotiation.cc:299] Negotiation successful
0807 22:59:45.148188 (+ 745us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260807 22:59:45.153230 15948 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ba7dd4fb2b04e91a64aeb796378871b: No bootstrap required, opened a new log
I20260807 22:59:45.156813 16214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:45.121897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34220 (local address 127.25.94.60:34111)
0807 22:59:45.122064 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0807 22:59:45.122072 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 22:59:45.131253 (+ 9181us) server_negotiation.cc:407] Connection header received
0807 22:59:45.132322 (+ 1069us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:59:45.132331 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:59:45.132473 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:59:45.132584 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:59:45.138115 (+ 5531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:59:45.139197 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:59:45.143197 (+ 4000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:59:45.144315 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:59:45.144501 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:59:45.148979 (+ 4478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:59:45.148989 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:59:45.148997 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:59:45.149049 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:59:45.153296 (+ 4247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:59:45.154278 (+ 982us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:59:45.154284 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:59:45.154287 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:59:45.154455 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:59:45.154897 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:59:45.154901 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:59:45.154904 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:59:45.155296 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:59:45.155518 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0807 22:59:45.156148 (+ 630us) server_negotiation.cc:299] Negotiation successful
0807 22:59:45.156413 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260807 22:59:45.157467 16083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:45.128109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38006 (local address 127.25.94.61:36505)
0807 22:59:45.128311 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0807 22:59:45.128317 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 22:59:45.133432 (+ 5115us) server_negotiation.cc:407] Connection header received
0807 22:59:45.134016 (+ 584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:59:45.134023 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:59:45.134129 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:59:45.134215 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:59:45.136771 (+ 2556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:59:45.138251 (+ 1480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:59:45.140376 (+ 2125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:59:45.141045 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:59:45.141155 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:59:45.146407 (+ 5252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:59:45.146414 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:59:45.146419 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:59:45.146466 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:59:45.152823 (+ 6357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:59:45.154020 (+ 1197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:59:45.154029 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:59:45.154035 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:59:45.154264 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:59:45.154737 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:59:45.154742 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:59:45.154745 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:59:45.155174 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:59:45.155430 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0807 22:59:45.156776 (+ 1346us) server_negotiation.cc:299] Negotiation successful
0807 22:59:45.157067 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260807 22:59:45.169507 15948 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4ba7dd4fb2b04e91a64aeb796378871b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ba7dd4fb2b04e91a64aeb796378871b" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 36419 } } peers { permanent_uuid: "f0cedc141ce141f5b930a1044ca76efa" member_type: VOTER last_known_addr { host: "127.25.94.61" port: 36505 } } peers { permanent_uuid: "b31b5e7eb77a404a99709ca318ecadf4" member_type: VOTER last_known_addr { host: "127.25.94.60" port: 34111 } }
I20260807 22:59:45.170315 15948 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4ba7dd4fb2b04e91a64aeb796378871b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:59:45.170523 15948 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4ba7dd4fb2b04e91a64aeb796378871b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ba7dd4fb2b04e91a64aeb796378871b, State: Initialized, Role: FOLLOWER
I20260807 22:59:45.171490 15948 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4ba7dd4fb2b04e91a64aeb796378871b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ba7dd4fb2b04e91a64aeb796378871b" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 36419 } } peers { permanent_uuid: "f0cedc141ce141f5b930a1044ca76efa" member_type: VOTER last_known_addr { host: "127.25.94.61" port: 36505 } } peers { permanent_uuid: "b31b5e7eb77a404a99709ca318ecadf4" member_type: VOTER last_known_addr { host: "127.25.94.60" port: 34111 } }
I20260807 22:59:45.176939 16258 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ba7dd4fb2b04e91a64aeb796378871b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ba7dd4fb2b04e91a64aeb796378871b" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 36419 } } peers { permanent_uuid: "f0cedc141ce141f5b930a1044ca76efa" member_type: VOTER last_known_addr { host: "127.25.94.61" port: 36505 } } peers { permanent_uuid: "b31b5e7eb77a404a99709ca318ecadf4" member_type: VOTER last_known_addr { host: "127.25.94.60" port: 34111 } } }
I20260807 22:59:45.177532 16258 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ba7dd4fb2b04e91a64aeb796378871b [sys.catalog]: This master's current role is: FOLLOWER
I20260807 22:59:45.178656 15948 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4ba7dd4fb2b04e91a64aeb796378871b [sys.catalog]: configured and running, proceeding with master startup.
W20260807 22:59:45.179906 16256 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b31b5e7eb77a404a99709ca318ecadf4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 22:59:45.180132 16256 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260807 22:59:45.183880 15859 tablet.cc:2369] T 00000000000000000000000000000000 P 4ba7dd4fb2b04e91a64aeb796378871b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 22:59:45.172578 16088 master_runner.cc:186] Error getting master registration for 127.25.94.62:36419: OK, instance_id { permanent_uuid: "4ba7dd4fb2b04e91a64aeb796378871b" instance_seqno: 1786143583944800 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260807 22:59:45.189728 16088 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260807 22:59:45.200091 16279 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4ba7dd4fb2b04e91a64aeb796378871b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 22:59:45.200410 16279 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 22:59:45.199174 15824 master_runner.cc:186] Error getting master registration for 127.25.94.61:36505: OK, instance_id { permanent_uuid: "f0cedc141ce141f5b930a1044ca76efa" instance_seqno: 1786143584494474 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260807 22:59:45.211947 15824 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260807 22:59:45.226119 16076 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.94.60" port: 34111 } attempt: 5
Aug 07 22:59:45 dist-test-slave-z8x9 krb5kdc[15800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143584, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.61@KRBTEST.COM for kudu/127.25.94.60@KRBTEST.COM
I20260807 22:59:45.247133 16080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:45.227058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.60:34111 (local address 127.0.0.1:34228)
0807 22:59:45.227375 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0807 22:59:45.227396 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 22:59:45.227534 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:59:45.227948 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:59:45.227954 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:59:45.228531 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:59:45.228754 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:59:45.228767 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:59:45.230804 (+ 2037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:59:45.230811 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:59:45.232259 (+ 1448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:59:45.232276 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:59:45.233392 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:59:45.233400 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:59:45.233607 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:59:45.234371 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:59:45.234399 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:59:45.237486 (+ 3087us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:59:45.243606 (+ 6120us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:59:45.243612 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:59:45.243616 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:59:45.244231 (+ 615us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:59:45.244769 (+ 538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:59:45.244774 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:59:45.244778 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:59:45.245020 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:59:45.245979 (+ 959us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:59:45.245986 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:59:45.246200 (+ 214us) client_negotiation.cc:769] Sending connection context
0807 22:59:45.246382 (+ 182us) client_negotiation.cc:240] Negotiation successful
0807 22:59:45.246726 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186}
I20260807 22:59:45.247295 16214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:45.227183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34228 (local address 127.25.94.60:34111)
0807 22:59:45.227388 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0807 22:59:45.227396 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 22:59:45.227502 (+ 106us) server_negotiation.cc:407] Connection header received
0807 22:59:45.227666 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:59:45.227673 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:59:45.227757 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:59:45.227877 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:59:45.228952 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:59:45.230171 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:59:45.232467 (+ 2296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:59:45.233206 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:59:45.233341 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:59:45.237738 (+ 4397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:59:45.237840 (+ 102us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:59:45.237847 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:59:45.237911 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:59:45.243414 (+ 5503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:59:45.244392 (+ 978us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:59:45.244399 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:59:45.244403 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:59:45.244617 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:59:45.245226 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:59:45.245232 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:59:45.245236 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:59:45.245688 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:59:45.245887 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0807 22:59:45.246529 (+ 642us) server_negotiation.cc:299] Negotiation successful
0807 22:59:45.246876 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260807 22:59:45.253961 16076 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0cedc141ce141f5b930a1044ca76efa: Bootstrap starting.
I20260807 22:59:45.260155 16076 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f0cedc141ce141f5b930a1044ca76efa: Neither blocks nor log segments found. Creating new log.
I20260807 22:59:45.261696 16076 log.cc:824] T 00000000000000000000000000000000 P f0cedc141ce141f5b930a1044ca76efa: Log is configured to *not* fsync() on all Append() calls
I20260807 22:59:45.265565 16076 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0cedc141ce141f5b930a1044ca76efa: No bootstrap required, opened a new log
W20260807 22:59:45.267952 16122 tablet.cc:2369] T 00000000000000000000000000000000 P b31b5e7eb77a404a99709ca318ecadf4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 22:59:45.277195 16076 raft_consensus.cc:348] T 00000000000000000000000000000000 P f0cedc141ce141f5b930a1044ca76efa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ba7dd4fb2b04e91a64aeb796378871b" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 36419 } } peers { permanent_uuid: "f0cedc141ce141f5b930a1044ca76efa" member_type: VOTER last_known_addr { host: "127.25.94.61" port: 36505 } } peers { permanent_uuid: "b31b5e7eb77a404a99709ca318ecadf4" member_type: VOTER last_known_addr { host: "127.25.94.60" port: 34111 } }
I20260807 22:59:45.277678 16076 raft_consensus.cc:374] T 00000000000000000000000000000000 P f0cedc141ce141f5b930a1044ca76efa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:59:45.277807 16076 raft_consensus.cc:729] T 00000000000000000000000000000000 P f0cedc141ce141f5b930a1044ca76efa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0cedc141ce141f5b930a1044ca76efa, State: Initialized, Role: FOLLOWER
I20260807 22:59:45.278542 16076 consensus_queue.cc:260] T 00000000000000000000000000000000 P f0cedc141ce141f5b930a1044ca76efa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ba7dd4fb2b04e91a64aeb796378871b" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 36419 } } peers { permanent_uuid: "f0cedc141ce141f5b930a1044ca76efa" member_type: VOTER last_known_addr { host: "127.25.94.61" port: 36505 } } peers { permanent_uuid: "b31b5e7eb77a404a99709ca318ecadf4" member_type: VOTER last_known_addr { host: "127.25.94.60" port: 34111 } }
I20260807 22:59:45.279724 16284 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0cedc141ce141f5b930a1044ca76efa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ba7dd4fb2b04e91a64aeb796378871b" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 36419 } } peers { permanent_uuid: "f0cedc141ce141f5b930a1044ca76efa" member_type: VOTER last_known_addr { host: "127.25.94.61" port: 36505 } } peers { permanent_uuid: "b31b5e7eb77a404a99709ca318ecadf4" member_type: VOTER last_known_addr { host: "127.25.94.60" port: 34111 } } }
I20260807 22:59:45.280251 16284 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0cedc141ce141f5b930a1044ca76efa [sys.catalog]: This master's current role is: FOLLOWER
I20260807 22:59:45.281136 16076 sys_catalog.cc:564] T 00000000000000000000000000000000 P f0cedc141ce141f5b930a1044ca76efa [sys.catalog]: configured and running, proceeding with master startup.
I20260807 22:59:45.295804 16258 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4ba7dd4fb2b04e91a64aeb796378871b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 22:59:45.296150 16258 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4ba7dd4fb2b04e91a64aeb796378871b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ba7dd4fb2b04e91a64aeb796378871b" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 36419 } } peers { permanent_uuid: "f0cedc141ce141f5b930a1044ca76efa" member_type: VOTER last_known_addr { host: "127.25.94.61" port: 36505 } } peers { permanent_uuid: "b31b5e7eb77a404a99709ca318ecadf4" member_type: VOTER last_known_addr { host: "127.25.94.60" port: 34111 } }
I20260807 22:59:45.297964 16258 leader_election.cc:288] T 00000000000000000000000000000000 P 4ba7dd4fb2b04e91a64aeb796378871b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f0cedc141ce141f5b930a1044ca76efa (127.25.94.61:36505), b31b5e7eb77a404a99709ca318ecadf4 (127.25.94.60:34111)
I20260807 22:59:45.299150 16163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4ba7dd4fb2b04e91a64aeb796378871b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b31b5e7eb77a404a99709ca318ecadf4" is_pre_election: true
I20260807 22:59:45.299813 16163 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b31b5e7eb77a404a99709ca318ecadf4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ba7dd4fb2b04e91a64aeb796378871b in term 0.
I20260807 22:59:45.301100 16028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4ba7dd4fb2b04e91a64aeb796378871b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0cedc141ce141f5b930a1044ca76efa" is_pre_election: true
I20260807 22:59:45.301710 16028 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f0cedc141ce141f5b930a1044ca76efa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ba7dd4fb2b04e91a64aeb796378871b in term 0.
I20260807 22:59:45.302893 15854 leader_election.cc:302] T 00000000000000000000000000000000 P 4ba7dd4fb2b04e91a64aeb796378871b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ba7dd4fb2b04e91a64aeb796378871b, b31b5e7eb77a404a99709ca318ecadf4; no voters:
I20260807 22:59:45.306646 16258 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4ba7dd4fb2b04e91a64aeb796378871b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 22:59:45.306885 16258 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4ba7dd4fb2b04e91a64aeb796378871b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 22:59:45.307004 16258 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4ba7dd4fb2b04e91a64aeb796378871b [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:59:45.310667 16258 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4ba7dd4fb2b04e91a64aeb796378871b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ba7dd4fb2b04e91a64aeb796378871b" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 36419 } } peers { permanent_uuid: "f0cedc141ce141f5b930a1044ca76efa" member_type: VOTER last_known_addr { host: "127.25.94.61" port: 36505 } } peers { permanent_uuid: "b31b5e7eb77a404a99709ca318ecadf4" member_type: VOTER last_known_addr { host: "127.25.94.60" port: 34111 } }
I20260807 22:59:45.311959 16028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4ba7dd4fb2b04e91a64aeb796378871b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0cedc141ce141f5b930a1044ca76efa"
I20260807 22:59:45.312211 16028 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f0cedc141ce141f5b930a1044ca76efa [term 0 FOLLOWER]: Advancing to term 1
W20260807 22:59:45.313910 16301 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f0cedc141ce141f5b930a1044ca76efa: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 22:59:45.314110 16301 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 22:59:45.315935 16028 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f0cedc141ce141f5b930a1044ca76efa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ba7dd4fb2b04e91a64aeb796378871b in term 1.
I20260807 22:59:45.319885 15958 master_runner.cc:263] No leader master found from master f0cedc141ce141f5b930a1044ca76efa
I20260807 22:59:45.320034 15958 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260807 22:59:45.320428 15856 leader_election.cc:302] T 00000000000000000000000000000000 P 4ba7dd4fb2b04e91a64aeb796378871b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ba7dd4fb2b04e91a64aeb796378871b, f0cedc141ce141f5b930a1044ca76efa; no voters:
I20260807 22:59:45.323974 16163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4ba7dd4fb2b04e91a64aeb796378871b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b31b5e7eb77a404a99709ca318ecadf4"
I20260807 22:59:45.324234 16163 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b31b5e7eb77a404a99709ca318ecadf4 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:59:45.327857 16163 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b31b5e7eb77a404a99709ca318ecadf4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ba7dd4fb2b04e91a64aeb796378871b in term 1.
I20260807 22:59:45.328927 16258 leader_election.cc:288] T 00000000000000000000000000000000 P 4ba7dd4fb2b04e91a64aeb796378871b [CANDIDATE]: Term 1 election: Requested vote from peers f0cedc141ce141f5b930a1044ca76efa (127.25.94.61:36505), b31b5e7eb77a404a99709ca318ecadf4 (127.25.94.60:34111)
I20260807 22:59:45.329241 16258 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4ba7dd4fb2b04e91a64aeb796378871b [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:59:45.329417 16258 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4ba7dd4fb2b04e91a64aeb796378871b [term 1 LEADER]: Becoming Leader. State: Replica: 4ba7dd4fb2b04e91a64aeb796378871b, State: Running, Role: LEADER
I20260807 22:59:45.330061 16258 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4ba7dd4fb2b04e91a64aeb796378871b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ba7dd4fb2b04e91a64aeb796378871b" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 36419 } } peers { permanent_uuid: "f0cedc141ce141f5b930a1044ca76efa" member_type: VOTER last_known_addr { host: "127.25.94.61" port: 36505 } } peers { permanent_uuid: "b31b5e7eb77a404a99709ca318ecadf4" member_type: VOTER last_known_addr { host: "127.25.94.60" port: 34111 } }
I20260807 22:59:45.345860 16306 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ba7dd4fb2b04e91a64aeb796378871b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4ba7dd4fb2b04e91a64aeb796378871b. Latest consensus state: current_term: 1 leader_uuid: "4ba7dd4fb2b04e91a64aeb796378871b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ba7dd4fb2b04e91a64aeb796378871b" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 36419 } } peers { permanent_uuid: "f0cedc141ce141f5b930a1044ca76efa" member_type: VOTER last_known_addr { host: "127.25.94.61" port: 36505 } } peers { permanent_uuid: "b31b5e7eb77a404a99709ca318ecadf4" member_type: VOTER last_known_addr { host: "127.25.94.60" port: 34111 } } }
I20260807 22:59:45.346372 16306 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ba7dd4fb2b04e91a64aeb796378871b [sys.catalog]: This master's current role is: LEADER
I20260807 22:59:45.353811 16310 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 22:59:45.356132 16310 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 22:59:45.372211 16163 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b31b5e7eb77a404a99709ca318ecadf4 [term 1 FOLLOWER]: Refusing update from remote peer 4ba7dd4fb2b04e91a64aeb796378871b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 22:59:45.373347 16306 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4ba7dd4fb2b04e91a64aeb796378871b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b31b5e7eb77a404a99709ca318ecadf4" member_type: VOTER last_known_addr { host: "127.25.94.60" port: 34111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 22:59:45.381418 16028 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f0cedc141ce141f5b930a1044ca76efa [term 1 FOLLOWER]: Refusing update from remote peer 4ba7dd4fb2b04e91a64aeb796378871b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 22:59:45.384326 16306 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4ba7dd4fb2b04e91a64aeb796378871b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0cedc141ce141f5b930a1044ca76efa" member_type: VOTER last_known_addr { host: "127.25.94.61" port: 36505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 22:59:45.418728 16284 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0cedc141ce141f5b930a1044ca76efa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4ba7dd4fb2b04e91a64aeb796378871b. Latest consensus state: current_term: 1 leader_uuid: "4ba7dd4fb2b04e91a64aeb796378871b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ba7dd4fb2b04e91a64aeb796378871b" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 36419 } } peers { permanent_uuid: "f0cedc141ce141f5b930a1044ca76efa" member_type: VOTER last_known_addr { host: "127.25.94.61" port: 36505 } } peers { permanent_uuid: "b31b5e7eb77a404a99709ca318ecadf4" member_type: VOTER last_known_addr { host: "127.25.94.60" port: 34111 } } }
I20260807 22:59:45.419077 16284 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0cedc141ce141f5b930a1044ca76efa [sys.catalog]: This master's current role is: FOLLOWER
I20260807 22:59:45.429628 16316 mvcc.cc:205] Tried to move back new op lower bound from 7316044125670834176 to 7316044125526056960. Current Snapshot: MvccSnapshot[applied={T|T < 7316044125670834176}]
I20260807 22:59:45.430531 16236 sys_catalog.cc:455] T 00000000000000000000000000000000 P b31b5e7eb77a404a99709ca318ecadf4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4ba7dd4fb2b04e91a64aeb796378871b. Latest consensus state: current_term: 1 leader_uuid: "4ba7dd4fb2b04e91a64aeb796378871b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ba7dd4fb2b04e91a64aeb796378871b" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 36419 } } peers { permanent_uuid: "f0cedc141ce141f5b930a1044ca76efa" member_type: VOTER last_known_addr { host: "127.25.94.61" port: 36505 } } peers { permanent_uuid: "b31b5e7eb77a404a99709ca318ecadf4" member_type: VOTER last_known_addr { host: "127.25.94.60" port: 34111 } } }
I20260807 22:59:45.430961 16236 sys_catalog.cc:458] T 00000000000000000000000000000000 P b31b5e7eb77a404a99709ca318ecadf4 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 22:59:45.444097 16284 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0cedc141ce141f5b930a1044ca76efa [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4ba7dd4fb2b04e91a64aeb796378871b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ba7dd4fb2b04e91a64aeb796378871b" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 36419 } } peers { permanent_uuid: "f0cedc141ce141f5b930a1044ca76efa" member_type: VOTER last_known_addr { host: "127.25.94.61" port: 36505 } } peers { permanent_uuid: "b31b5e7eb77a404a99709ca318ecadf4" member_type: VOTER last_known_addr { host: "127.25.94.60" port: 34111 } } }
I20260807 22:59:45.444412 16284 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0cedc141ce141f5b930a1044ca76efa [sys.catalog]: This master's current role is: FOLLOWER
I20260807 22:59:45.445770 16236 sys_catalog.cc:455] T 00000000000000000000000000000000 P b31b5e7eb77a404a99709ca318ecadf4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4ba7dd4fb2b04e91a64aeb796378871b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ba7dd4fb2b04e91a64aeb796378871b" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 36419 } } peers { permanent_uuid: "f0cedc141ce141f5b930a1044ca76efa" member_type: VOTER last_known_addr { host: "127.25.94.61" port: 36505 } } peers { permanent_uuid: "b31b5e7eb77a404a99709ca318ecadf4" member_type: VOTER last_known_addr { host: "127.25.94.60" port: 34111 } } }
I20260807 22:59:45.446038 16236 sys_catalog.cc:458] T 00000000000000000000000000000000 P b31b5e7eb77a404a99709ca318ecadf4 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 22:59:45.451715 16310 catalog_manager.cc:1347] Generated new cluster ID: 2111c578c88f4f3c99e5af9d3edc3596
I20260807 22:59:45.451954 16310 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 22:59:45.454743 16306 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ba7dd4fb2b04e91a64aeb796378871b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4ba7dd4fb2b04e91a64aeb796378871b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ba7dd4fb2b04e91a64aeb796378871b" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 36419 } } peers { permanent_uuid: "f0cedc141ce141f5b930a1044ca76efa" member_type: VOTER last_known_addr { host: "127.25.94.61" port: 36505 } } peers { permanent_uuid: "b31b5e7eb77a404a99709ca318ecadf4" member_type: VOTER last_known_addr { host: "127.25.94.60" port: 34111 } } }
I20260807 22:59:45.455041 16306 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ba7dd4fb2b04e91a64aeb796378871b [sys.catalog]: This master's current role is: LEADER
I20260807 22:59:45.455323 16306 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ba7dd4fb2b04e91a64aeb796378871b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4ba7dd4fb2b04e91a64aeb796378871b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ba7dd4fb2b04e91a64aeb796378871b" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 36419 } } peers { permanent_uuid: "f0cedc141ce141f5b930a1044ca76efa" member_type: VOTER last_known_addr { host: "127.25.94.61" port: 36505 } } peers { permanent_uuid: "b31b5e7eb77a404a99709ca318ecadf4" member_type: VOTER last_known_addr { host: "127.25.94.60" port: 34111 } } }
I20260807 22:59:45.455554 16306 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ba7dd4fb2b04e91a64aeb796378871b [sys.catalog]: This master's current role is: LEADER
I20260807 22:59:45.500605 16310 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 22:59:45.502462 16310 catalog_manager.cc:1504] Loading token signing keys...
I20260807 22:59:45.526060 16310 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4ba7dd4fb2b04e91a64aeb796378871b: Generated new TSK 0
I20260807 22:59:45.527163 16310 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 22:59:46.182523 16256 catalog_manager.cc:1259] Loaded cluster ID: 2111c578c88f4f3c99e5af9d3edc3596
I20260807 22:59:46.182648 16256 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b31b5e7eb77a404a99709ca318ecadf4: loading cluster ID for follower catalog manager: success
I20260807 22:59:46.185299 16256 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b31b5e7eb77a404a99709ca318ecadf4: acquiring CA information for follower catalog manager: success
I20260807 22:59:46.187495 16256 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b31b5e7eb77a404a99709ca318ecadf4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260807 22:59:46.316356 16301 catalog_manager.cc:1259] Loaded cluster ID: 2111c578c88f4f3c99e5af9d3edc3596
I20260807 22:59:46.316538 16301 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f0cedc141ce141f5b930a1044ca76efa: loading cluster ID for follower catalog manager: success
I20260807 22:59:46.319113 16301 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f0cedc141ce141f5b930a1044ca76efa: acquiring CA information for follower catalog manager: success
I20260807 22:59:46.320930 16301 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f0cedc141ce141f5b930a1044ca76efa: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260807 22:59:46.694015 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 15824
I20260807 22:59:46.695448 15936 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 22:59:46.805708 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 15824
I20260807 22:59:46.835659 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 15958
I20260807 22:59:46.836975 16066 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 22:59:46.944073 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 15958
I20260807 22:59:46.970764 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 16088
I20260807 22:59:46.971915 16199 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 22:59:47.080035 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 16088
2026-08-07T22:59:47Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (5264 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786143361779964-25976-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 22:59:47 dist-test-slave-z8x9 krb5kdc[16344](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 22:59:47 dist-test-slave-z8x9 krb5kdc[16344](info): set up 2 sockets
Aug 07 22:59:47 dist-test-slave-z8x9 krb5kdc[16344](info): commencing operation
krb5kdc: starting...
W20260807 22:59:48.460453 25976 mini_kdc.cc:121] Time spent starting KDC: real 1.335s user 0.001s sys 0.036s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786143361779964-25976-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786143361779964-25976-0/krb5kdc/test-user.keytab.
Aug 07 22:59:48 dist-test-slave-z8x9 krb5kdc[16344](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143588, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T22:59:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T22:59:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.94.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.62.keytab.
Entry for principal kudu/127.25.94.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.62.keytab.
I20260807 22:59:48.758136 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786143361779964-25976-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786143361779964-25976-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786143361779964-25976-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.62:35023
--webserver_interface=127.25.94.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.62.keytab
--principal=kudu/127.25.94.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.20:38279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.94.62:35023
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786143361779964-25976-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786143361779964-25976-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786143361779964-25976-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:59:49.105069 16374 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:59:49.105494 16374 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:59:49.105687 16374 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:59:49.113294 16374 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 22:59:49.113504 16374 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:59:49.113613 16374 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:59:49.113754 16374 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 22:59:49.113807 16374 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 22:59:49.124229 16374 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:38279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786143361779964-25976-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786143361779964-25976-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.94.62:35023
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.62.keytab
--principal=kudu/127.25.94.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.62:35023
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.94.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786143361779964-25976-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 22:59:49.125926 16374 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:59:49.127385 16374 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:59:49.140875 16381 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:59:49.140849 16386 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:59:49.141106 16380 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:59:49.141765 16374 server_base.cc:1034] running on GCE node
I20260807 22:59:49.142673 16374 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:59:49.144192 16374 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:59:49.145468 16374 hybrid_clock.cc:648] HybridClock initialized: now 1786143589145402 us; error 54 us; skew 500 ppm
I20260807 22:59:49.145865 16374 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 22:59:49 dist-test-slave-z8x9 krb5kdc[16344](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143589, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:59:49.153036 16374 init.cc:377] Logged in from keytab as kudu/127.25.94.62@KRBTEST.COM (short username kudu)
I20260807 22:59:49.154907 16374 webserver.cc:460] Webserver started at http://127.25.94.62:46515/ using document root <none> and password file <none>
I20260807 22:59:49.155687 16374 fs_manager.cc:359] Metadata directory not provided
I20260807 22:59:49.155875 16374 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:59:49.156234 16374 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:59:49.159608 16374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786143361779964-25976-0/minicluster-data/master-0/data/instance:
uuid: "019c4262649a407d816d23a07608ec0b"
format_stamp: "Formatted at 2026-08-07 22:59:49 on dist-test-slave-z8x9"
server_key: "595c62f9a9cf46b9a1a361069b0f581c"
server_key_iv: "35eaa210bcad7587b1dbeb74bc3e8161"
server_key_version: "encryptionkey@0"
I20260807 22:59:49.160766 16374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786143361779964-25976-0/minicluster-data/master-0/wal/instance:
uuid: "019c4262649a407d816d23a07608ec0b"
format_stamp: "Formatted at 2026-08-07 22:59:49 on dist-test-slave-z8x9"
server_key: "595c62f9a9cf46b9a1a361069b0f581c"
server_key_iv: "35eaa210bcad7587b1dbeb74bc3e8161"
server_key_version: "encryptionkey@0"
I20260807 22:59:49.167273 16374 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.004s
I20260807 22:59:49.171387 16400 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:59:49.172976 16374 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 22:59:49.173297 16374 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786143361779964-25976-0/minicluster-data/master-0/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786143361779964-25976-0/minicluster-data/master-0/wal
uuid: "019c4262649a407d816d23a07608ec0b"
format_stamp: "Formatted at 2026-08-07 22:59:49 on dist-test-slave-z8x9"
server_key: "595c62f9a9cf46b9a1a361069b0f581c"
server_key_iv: "35eaa210bcad7587b1dbeb74bc3e8161"
server_key_version: "encryptionkey@0"
I20260807 22:59:49.173535 16374 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786143361779964-25976-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786143361779964-25976-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786143361779964-25976-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:59:49.202507 16374 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:59:49.207109 16374 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:59:49.207454 16374 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:59:49.228140 16374 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.62:35023
I20260807 22:59:49.228127 16480 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.62:35023 every 8 connection(s)
I20260807 22:59:49.229775 16374 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
I20260807 22:59:49.233824 16481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:59:49.238601 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 16374
I20260807 22:59:49.238867 25976 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786143361779964-25976-0/minicluster-data/master-0/wal/instance
I20260807 22:59:49.239583 25976 external_mini_cluster.cc:1428] Setting key 737648d383e56c938b894b2cb1257236
I20260807 22:59:49.246209 16481 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 019c4262649a407d816d23a07608ec0b: Bootstrap starting.
I20260807 22:59:49.249807 16481 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 019c4262649a407d816d23a07608ec0b: Neither blocks nor log segments found. Creating new log.
I20260807 22:59:49.251116 16481 log.cc:824] T 00000000000000000000000000000000 P 019c4262649a407d816d23a07608ec0b: Log is configured to *not* fsync() on all Append() calls
I20260807 22:59:49.254793 16481 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 019c4262649a407d816d23a07608ec0b: No bootstrap required, opened a new log
Aug 07 22:59:49 dist-test-slave-z8x9 krb5kdc[16344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143588, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.62@KRBTEST.COM
I20260807 22:59:49.264343 16481 raft_consensus.cc:348] T 00000000000000000000000000000000 P 019c4262649a407d816d23a07608ec0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "019c4262649a407d816d23a07608ec0b" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 35023 } }
I20260807 22:59:49.264811 16481 raft_consensus.cc:374] T 00000000000000000000000000000000 P 019c4262649a407d816d23a07608ec0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:59:49.264976 16481 raft_consensus.cc:729] T 00000000000000000000000000000000 P 019c4262649a407d816d23a07608ec0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 019c4262649a407d816d23a07608ec0b, State: Initialized, Role: FOLLOWER
I20260807 22:59:49.265655 16481 consensus_queue.cc:260] T 00000000000000000000000000000000 P 019c4262649a407d816d23a07608ec0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "019c4262649a407d816d23a07608ec0b" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 35023 } }
I20260807 22:59:49.265955 16481 raft_consensus.cc:388] T 00000000000000000000000000000000 P 019c4262649a407d816d23a07608ec0b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 22:59:49.266115 16481 raft_consensus.cc:482] T 00000000000000000000000000000000 P 019c4262649a407d816d23a07608ec0b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 22:59:49.266326 16481 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 019c4262649a407d816d23a07608ec0b [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:59:49.266364 16484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:49.241760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51384 (local address 127.25.94.62:35023)
0807 22:59:49.242719 (+ 959us) server_negotiation.cc:206] Beginning negotiation
0807 22:59:49.242735 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0807 22:59:49.242793 (+ 58us) server_negotiation.cc:407] Connection header received
0807 22:59:49.244799 (+ 2006us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:59:49.244827 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:59:49.245373 (+ 546us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:59:49.245828 (+ 455us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:59:49.247491 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:59:49.248948 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:59:49.250568 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:59:49.251380 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:59:49.251592 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:59:49.256867 (+ 5275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:59:49.256876 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:59:49.256893 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:59:49.256941 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:59:49.260567 (+ 3626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:59:49.261795 (+ 1228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:59:49.261808 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:59:49.261815 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:59:49.262073 (+ 258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:59:49.262567 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:59:49.262571 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:59:49.262573 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:59:49.263205 (+ 632us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:59:49.263447 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0807 22:59:49.264051 (+ 604us) server_negotiation.cc:299] Negotiation successful
0807 22:59:49.264880 (+ 829us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":713,"thread_start_us":468,"threads_started":1}
I20260807 22:59:49.269482 16481 raft_consensus.cc:504] T 00000000000000000000000000000000 P 019c4262649a407d816d23a07608ec0b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "019c4262649a407d816d23a07608ec0b" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 35023 } }
I20260807 22:59:49.270089 16481 leader_election.cc:302] T 00000000000000000000000000000000 P 019c4262649a407d816d23a07608ec0b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 019c4262649a407d816d23a07608ec0b; no voters:
I20260807 22:59:49.270792 25976 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 22:59:49.270809 16481 leader_election.cc:288] T 00000000000000000000000000000000 P 019c4262649a407d816d23a07608ec0b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 22:59:49.270941 16487 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 019c4262649a407d816d23a07608ec0b [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:59:49.272192 16487 raft_consensus.cc:686] T 00000000000000000000000000000000 P 019c4262649a407d816d23a07608ec0b [term 1 LEADER]: Becoming Leader. State: Replica: 019c4262649a407d816d23a07608ec0b, State: Running, Role: LEADER
I20260807 22:59:49.272809 16487 consensus_queue.cc:237] T 00000000000000000000000000000000 P 019c4262649a407d816d23a07608ec0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "019c4262649a407d816d23a07608ec0b" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 35023 } }
I20260807 22:59:49.272984 16481 sys_catalog.cc:564] T 00000000000000000000000000000000 P 019c4262649a407d816d23a07608ec0b [sys.catalog]: configured and running, proceeding with master startup.
I20260807 22:59:49.275909 16488 sys_catalog.cc:455] T 00000000000000000000000000000000 P 019c4262649a407d816d23a07608ec0b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "019c4262649a407d816d23a07608ec0b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "019c4262649a407d816d23a07608ec0b" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 35023 } } }
I20260807 22:59:49.276274 16489 sys_catalog.cc:455] T 00000000000000000000000000000000 P 019c4262649a407d816d23a07608ec0b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 019c4262649a407d816d23a07608ec0b. Latest consensus state: current_term: 1 leader_uuid: "019c4262649a407d816d23a07608ec0b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "019c4262649a407d816d23a07608ec0b" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 35023 } } }
I20260807 22:59:49.276582 16488 sys_catalog.cc:458] T 00000000000000000000000000000000 P 019c4262649a407d816d23a07608ec0b [sys.catalog]: This master's current role is: LEADER
I20260807 22:59:49.276582 16489 sys_catalog.cc:458] T 00000000000000000000000000000000 P 019c4262649a407d816d23a07608ec0b [sys.catalog]: This master's current role is: LEADER
I20260807 22:59:49.279529 16497 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 22:59:49.285094 16497 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 22:59:49.294257 16497 catalog_manager.cc:1347] Generated new cluster ID: dcd172d291d441f4a1afbc1b9d5e1a38
I20260807 22:59:49.294400 16497 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 22:59:49.306672 16497 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 22:59:49.308055 16497 catalog_manager.cc:1504] Loading token signing keys...
I20260807 22:59:49.313958 16497 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 019c4262649a407d816d23a07608ec0b: Generated new TSK 0
I20260807 22:59:49.314786 16497 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 22:59:49.333377 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 16374
I20260807 22:59:49.334657 16475 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 22:59:49.428761 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 16374
2026-08-07T22:59:49Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2347 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 22:59:49 dist-test-slave-z8x9 krb5kdc[16525](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 22:59:49 dist-test-slave-z8x9 krb5kdc[16525](info): set up 2 sockets
Aug 07 22:59:49 dist-test-slave-z8x9 krb5kdc[16525](info): commencing operation
krb5kdc: starting...
W20260807 22:59:50.795099 25976 mini_kdc.cc:121] Time spent starting KDC: real 1.322s user 0.000s sys 0.036s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/krb5kdc/test-user.keytab.
Aug 07 22:59:50 dist-test-slave-z8x9 krb5kdc[16525](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143590, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T22:59:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T22:59:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.94.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.62.keytab.
Entry for principal kudu/127.25.94.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.62.keytab.
I20260807 22:59:51.086123 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.62:42307
--webserver_interface=127.25.94.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.62.keytab
--principal=kudu/127.25.94.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.20:41671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.94.62:42307
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:59:51.444443 16546 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:59:51.444780 16546 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:59:51.444952 16546 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:59:51.452443 16546 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 22:59:51.452574 16546 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:59:51.452672 16546 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:59:51.452767 16546 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 22:59:51.452855 16546 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 22:59:51.463201 16546 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:41671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.94.62:42307
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.62.keytab
--principal=kudu/127.25.94.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.62:42307
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.94.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 22:59:51.464627 16546 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:59:51.466022 16546 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:59:51.477720 16556 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:59:51.477775 16562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:59:51.478323 16554 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:59:51.478713 16546 server_base.cc:1034] running on GCE node
I20260807 22:59:51.479537 16546 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:59:51.480895 16546 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:59:51.482133 16546 hybrid_clock.cc:648] HybridClock initialized: now 1786143591482048 us; error 101 us; skew 500 ppm
I20260807 22:59:51.482498 16546 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 22:59:51 dist-test-slave-z8x9 krb5kdc[16525](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143591, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:59:51.485924 16546 init.cc:377] Logged in from keytab as kudu/127.25.94.62@KRBTEST.COM (short username kudu)
I20260807 22:59:51.487521 16546 webserver.cc:460] Webserver started at http://127.25.94.62:42987/ using document root <none> and password file <none>
I20260807 22:59:51.488173 16546 fs_manager.cc:359] Metadata directory not provided
I20260807 22:59:51.488279 16546 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:59:51.488531 16546 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:59:51.491575 16546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/master-0/data/instance:
uuid: "70cb141317c44ced96448d499c69dd93"
format_stamp: "Formatted at 2026-08-07 22:59:51 on dist-test-slave-z8x9"
server_key: "a88c9b3689a8ca5be384dd3b9d50679a"
server_key_iv: "81694a9a1d3615340b77eca7ccef72cd"
server_key_version: "encryptionkey@0"
I20260807 22:59:51.492311 16546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/master-0/wal/instance:
uuid: "70cb141317c44ced96448d499c69dd93"
format_stamp: "Formatted at 2026-08-07 22:59:51 on dist-test-slave-z8x9"
server_key: "a88c9b3689a8ca5be384dd3b9d50679a"
server_key_iv: "81694a9a1d3615340b77eca7ccef72cd"
server_key_version: "encryptionkey@0"
I20260807 22:59:51.497941 16546 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260807 22:59:51.501948 16575 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:59:51.503472 16546 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 22:59:51.503628 16546 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/master-0/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/master-0/wal
uuid: "70cb141317c44ced96448d499c69dd93"
format_stamp: "Formatted at 2026-08-07 22:59:51 on dist-test-slave-z8x9"
server_key: "a88c9b3689a8ca5be384dd3b9d50679a"
server_key_iv: "81694a9a1d3615340b77eca7ccef72cd"
server_key_version: "encryptionkey@0"
I20260807 22:59:51.503820 16546 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:59:51.521394 16546 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:59:51.526229 16546 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:59:51.526484 16546 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:59:51.546056 16546 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.62:42307
I20260807 22:59:51.546077 16652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.62:42307 every 8 connection(s)
I20260807 22:59:51.547524 16546 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
I20260807 22:59:51.551299 16653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:59:51.556456 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 16546
I20260807 22:59:51.556694 25976 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/master-0/wal/instance
I20260807 22:59:51.557519 25976 external_mini_cluster.cc:1428] Setting key 82a6b11ca382e071c9aef711b77a4db0
I20260807 22:59:51.562183 16653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70cb141317c44ced96448d499c69dd93: Bootstrap starting.
I20260807 22:59:51.565898 16653 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 70cb141317c44ced96448d499c69dd93: Neither blocks nor log segments found. Creating new log.
I20260807 22:59:51.567107 16653 log.cc:824] T 00000000000000000000000000000000 P 70cb141317c44ced96448d499c69dd93: Log is configured to *not* fsync() on all Append() calls
I20260807 22:59:51.570415 16653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70cb141317c44ced96448d499c69dd93: No bootstrap required, opened a new log
Aug 07 22:59:51 dist-test-slave-z8x9 krb5kdc[16525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143590, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.62@KRBTEST.COM
I20260807 22:59:51.578732 16653 raft_consensus.cc:348] T 00000000000000000000000000000000 P 70cb141317c44ced96448d499c69dd93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70cb141317c44ced96448d499c69dd93" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 42307 } }
I20260807 22:59:51.579123 16653 raft_consensus.cc:374] T 00000000000000000000000000000000 P 70cb141317c44ced96448d499c69dd93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:59:51.579257 16653 raft_consensus.cc:729] T 00000000000000000000000000000000 P 70cb141317c44ced96448d499c69dd93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70cb141317c44ced96448d499c69dd93, State: Initialized, Role: FOLLOWER
I20260807 22:59:51.579989 16653 consensus_queue.cc:260] T 00000000000000000000000000000000 P 70cb141317c44ced96448d499c69dd93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70cb141317c44ced96448d499c69dd93" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 42307 } }
I20260807 22:59:51.580295 16653 raft_consensus.cc:388] T 00000000000000000000000000000000 P 70cb141317c44ced96448d499c69dd93 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 22:59:51.580443 16653 raft_consensus.cc:482] T 00000000000000000000000000000000 P 70cb141317c44ced96448d499c69dd93 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 22:59:51.580632 16653 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 70cb141317c44ced96448d499c69dd93 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:59:51.582796 16662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:51.559601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59782 (local address 127.25.94.62:42307)
0807 22:59:51.560486 (+ 885us) server_negotiation.cc:206] Beginning negotiation
0807 22:59:51.560497 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 22:59:51.560538 (+ 41us) server_negotiation.cc:407] Connection header received
0807 22:59:51.562254 (+ 1716us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:59:51.562279 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:59:51.562697 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:59:51.563007 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:59:51.564910 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:59:51.566377 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:59:51.568199 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:59:51.568970 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:59:51.569100 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:59:51.574275 (+ 5175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:59:51.574284 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:59:51.574301 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:59:51.574345 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:59:51.577785 (+ 3440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:59:51.578593 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:59:51.578600 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:59:51.578606 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:59:51.578779 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:59:51.579205 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:59:51.579212 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:59:51.579215 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:59:51.580029 (+ 814us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:59:51.580356 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0807 22:59:51.580877 (+ 521us) server_negotiation.cc:299] Negotiation successful
0807 22:59:51.581384 (+ 507us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":630,"thread_start_us":405,"threads_started":1}
I20260807 22:59:51.584134 16653 raft_consensus.cc:504] T 00000000000000000000000000000000 P 70cb141317c44ced96448d499c69dd93 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70cb141317c44ced96448d499c69dd93" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 42307 } }
I20260807 22:59:51.584662 16653 leader_election.cc:302] T 00000000000000000000000000000000 P 70cb141317c44ced96448d499c69dd93 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 70cb141317c44ced96448d499c69dd93; no voters:
I20260807 22:59:51.585374 16653 leader_election.cc:288] T 00000000000000000000000000000000 P 70cb141317c44ced96448d499c69dd93 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 22:59:51.585536 16664 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 70cb141317c44ced96448d499c69dd93 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:59:51.586621 16664 raft_consensus.cc:686] T 00000000000000000000000000000000 P 70cb141317c44ced96448d499c69dd93 [term 1 LEADER]: Becoming Leader. State: Replica: 70cb141317c44ced96448d499c69dd93, State: Running, Role: LEADER
I20260807 22:59:51.587165 16664 consensus_queue.cc:237] T 00000000000000000000000000000000 P 70cb141317c44ced96448d499c69dd93 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70cb141317c44ced96448d499c69dd93" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 42307 } }
I20260807 22:59:51.587450 16653 sys_catalog.cc:564] T 00000000000000000000000000000000 P 70cb141317c44ced96448d499c69dd93 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 22:59:51.590332 16665 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70cb141317c44ced96448d499c69dd93 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "70cb141317c44ced96448d499c69dd93" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70cb141317c44ced96448d499c69dd93" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 42307 } } }
I20260807 22:59:51.590306 16666 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70cb141317c44ced96448d499c69dd93 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 70cb141317c44ced96448d499c69dd93. Latest consensus state: current_term: 1 leader_uuid: "70cb141317c44ced96448d499c69dd93" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70cb141317c44ced96448d499c69dd93" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 42307 } } }
I20260807 22:59:51.590777 16665 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70cb141317c44ced96448d499c69dd93 [sys.catalog]: This master's current role is: LEADER
I20260807 22:59:51.590786 16666 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70cb141317c44ced96448d499c69dd93 [sys.catalog]: This master's current role is: LEADER
I20260807 22:59:51.591583 16677 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 22:59:51.598577 16677 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 22:59:51.611328 16677 catalog_manager.cc:1347] Generated new cluster ID: 3e96fff9e7a54bfabf8a7aca3984e5c7
I20260807 22:59:51.611466 16677 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 22:59:51.635737 16677 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 22:59:51.637351 16677 catalog_manager.cc:1504] Loading token signing keys...
I20260807 22:59:51.650424 16677 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 70cb141317c44ced96448d499c69dd93: Generated new TSK 0
I20260807 22:59:51.651280 16677 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.94.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.1.keytab.
Entry for principal kudu/127.25.94.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.1.keytab.
I20260807 22:59:51.724615 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.1:0
--local_ip_for_outbound_sockets=127.25.94.1
--webserver_interface=127.25.94.1
--webserver_port=0
--tserver_master_addrs=127.25.94.62:42307
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.1.keytab
--principal=kudu/127.25.94.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.20:41671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:59:52.066437 16694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:59:52.066736 16694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:59:52.066901 16694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:59:52.074412 16694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:59:52.074519 16694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:59:52.074729 16694 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.1
I20260807 22:59:52.084868 16694 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:41671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.1.keytab
--principal=kudu/127.25.94.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.94.1
--webserver_port=0
--tserver_master_addrs=127.25.94.62:42307
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.1
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 22:59:52.086220 16694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:59:52.087635 16694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:59:52.101856 16704 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:59:52.102005 16702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:59:52.102476 16708 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:59:52.102885 16694 server_base.cc:1034] running on GCE node
I20260807 22:59:52.103631 16694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:59:52.104686 16694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:59:52.105940 16694 hybrid_clock.cc:648] HybridClock initialized: now 1786143592105885 us; error 49 us; skew 500 ppm
I20260807 22:59:52.106333 16694 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 22:59:52 dist-test-slave-z8x9 krb5kdc[16525](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143592, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:59:52.110291 16694 init.cc:377] Logged in from keytab as kudu/127.25.94.1@KRBTEST.COM (short username kudu)
I20260807 22:59:52.111999 16694 webserver.cc:460] Webserver started at http://127.25.94.1:36475/ using document root <none> and password file <none>
I20260807 22:59:52.112777 16694 fs_manager.cc:359] Metadata directory not provided
I20260807 22:59:52.112949 16694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:59:52.113338 16694 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:59:52.116382 16694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-0/data/instance:
uuid: "cd086879c54a40bdb4dfecda24a0736b"
format_stamp: "Formatted at 2026-08-07 22:59:52 on dist-test-slave-z8x9"
server_key: "bd0710249b124060f98c5a18ebee52b4"
server_key_iv: "b38ebd9c39be82327b298dc096e71404"
server_key_version: "encryptionkey@0"
I20260807 22:59:52.117328 16694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-0/wal/instance:
uuid: "cd086879c54a40bdb4dfecda24a0736b"
format_stamp: "Formatted at 2026-08-07 22:59:52 on dist-test-slave-z8x9"
server_key: "bd0710249b124060f98c5a18ebee52b4"
server_key_iv: "b38ebd9c39be82327b298dc096e71404"
server_key_version: "encryptionkey@0"
I20260807 22:59:52.122958 16694 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260807 22:59:52.126842 16716 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:59:52.128328 16694 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260807 22:59:52.128547 16694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-0/wal
uuid: "cd086879c54a40bdb4dfecda24a0736b"
format_stamp: "Formatted at 2026-08-07 22:59:52 on dist-test-slave-z8x9"
server_key: "bd0710249b124060f98c5a18ebee52b4"
server_key_iv: "b38ebd9c39be82327b298dc096e71404"
server_key_version: "encryptionkey@0"
I20260807 22:59:52.128779 16694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:59:52.143390 16694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:59:52.148438 16694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:59:52.148903 16694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:59:52.150188 16694 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:59:52.151732 16694 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:59:52.151847 16694 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:59:52.152010 16694 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:59:52.152094 16694 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:59:52.188665 16694 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.1:46643
I20260807 22:59:52.188750 16895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.1:46643 every 8 connection(s)
I20260807 22:59:52.190150 16694 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-0/data/info.pb
I20260807 22:59:52.193946 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 16694
I20260807 22:59:52.194228 25976 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-0/wal/instance
I20260807 22:59:52.195154 25976 external_mini_cluster.cc:1428] Setting key 972d3a0eb1386a4ad3a67032c1c4789e
Aug 07 22:59:52 dist-test-slave-z8x9 krb5kdc[16525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143592, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.1@KRBTEST.COM for kudu/127.25.94.62@KRBTEST.COM
I20260807 22:59:52.215556 16900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:52.193377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.1:43893 (local address 127.25.94.62:42307)
0807 22:59:52.193942 (+ 565us) server_negotiation.cc:206] Beginning negotiation
0807 22:59:52.193949 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 22:59:52.195099 (+ 1150us) server_negotiation.cc:407] Connection header received
0807 22:59:52.197666 (+ 2567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:59:52.197684 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:59:52.197766 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:59:52.197884 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:59:52.199520 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:59:52.200695 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:59:52.202794 (+ 2099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:59:52.203555 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:59:52.203723 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:59:52.207827 (+ 4104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:59:52.207837 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:59:52.207845 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:59:52.207895 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:59:52.211503 (+ 3608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:59:52.212771 (+ 1268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:59:52.212776 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:59:52.212779 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:59:52.212932 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:59:52.213574 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:59:52.213578 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:59:52.213580 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:59:52.213872 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:59:52.213997 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0807 22:59:52.215062 (+ 1065us) server_negotiation.cc:299] Negotiation successful
0807 22:59:52.215285 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":277,"threads_started":1}
I20260807 22:59:52.217283 16901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:52.193849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:42307 (local address 127.25.94.1:43893)
0807 22:59:52.194869 (+ 1020us) negotiation.cc:107] Waiting for socket to connect
0807 22:59:52.194936 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0807 22:59:52.197222 (+ 2286us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:59:52.198120 (+ 898us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:59:52.198129 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:59:52.198712 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:59:52.199321 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:59:52.199340 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:59:52.200886 (+ 1546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:59:52.200891 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:59:52.202577 (+ 1686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:59:52.202622 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:59:52.203816 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:59:52.203822 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:59:52.204046 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:59:52.204847 (+ 801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:59:52.204872 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:59:52.207552 (+ 2680us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:59:52.211724 (+ 4172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:59:52.211738 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:59:52.211761 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:59:52.212556 (+ 795us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:59:52.213064 (+ 508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:59:52.213072 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:59:52.213079 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:59:52.213427 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:59:52.214020 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:59:52.214032 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:59:52.214556 (+ 524us) client_negotiation.cc:769] Sending connection context
0807 22:59:52.214889 (+ 333us) client_negotiation.cc:240] Negotiation successful
0807 22:59:52.215570 (+ 681us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":666,"thread_start_us":395,"threads_started":1}
I20260807 22:59:52.219331 16897 heartbeater.cc:347] Connected to a master server at 127.25.94.62:42307
I20260807 22:59:52.219696 16897 heartbeater.cc:464] Registering TS with master...
I20260807 22:59:52.220510 16897 heartbeater.cc:511] Master 127.25.94.62:42307 requested a full tablet report, sending...
I20260807 22:59:52.223042 16600 ts_manager.cc:194] Registered new tserver with Master: cd086879c54a40bdb4dfecda24a0736b (127.25.94.1:46643)
I20260807 22:59:52.224565 16600 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.94.1@KRBTEST.COM'} at 127.25.94.1:43893
WARNING: no policy specified for kudu/127.25.94.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.2.keytab.
Entry for principal kudu/127.25.94.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.2.keytab.
I20260807 22:59:52.264250 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.2:0
--local_ip_for_outbound_sockets=127.25.94.2
--webserver_interface=127.25.94.2
--webserver_port=0
--tserver_master_addrs=127.25.94.62:42307
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.2.keytab
--principal=kudu/127.25.94.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.20:41671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:59:52.614471 16905 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:59:52.614842 16905 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:59:52.615018 16905 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:59:52.622509 16905 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:59:52.622658 16905 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:59:52.622848 16905 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.2
I20260807 22:59:52.633181 16905 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:41671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.2.keytab
--principal=kudu/127.25.94.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.94.2
--webserver_port=0
--tserver_master_addrs=127.25.94.62:42307
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.2
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 22:59:52.634536 16905 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:59:52.635921 16905 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:59:52.649366 16920 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:59:52.649438 16915 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:59:52.649405 16918 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:59:52.650344 16905 server_base.cc:1034] running on GCE node
I20260807 22:59:52.651038 16905 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:59:52.652040 16905 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:59:52.653298 16905 hybrid_clock.cc:648] HybridClock initialized: now 1786143592653244 us; error 47 us; skew 500 ppm
I20260807 22:59:52.653626 16905 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 22:59:52 dist-test-slave-z8x9 krb5kdc[16525](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143592, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:59:52.656845 16905 init.cc:377] Logged in from keytab as kudu/127.25.94.2@KRBTEST.COM (short username kudu)
I20260807 22:59:52.658522 16905 webserver.cc:460] Webserver started at http://127.25.94.2:42255/ using document root <none> and password file <none>
I20260807 22:59:52.659184 16905 fs_manager.cc:359] Metadata directory not provided
I20260807 22:59:52.659291 16905 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:59:52.659536 16905 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:59:52.663980 16905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-1/data/instance:
uuid: "102b2e12e0f546e0a4f6402e8bd91481"
format_stamp: "Formatted at 2026-08-07 22:59:52 on dist-test-slave-z8x9"
server_key: "e857eaed6f47aa89a95c92b0d991ecae"
server_key_iv: "73e5cbde04ab8fe59499c8755f05edcd"
server_key_version: "encryptionkey@0"
I20260807 22:59:52.665148 16905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-1/wal/instance:
uuid: "102b2e12e0f546e0a4f6402e8bd91481"
format_stamp: "Formatted at 2026-08-07 22:59:52 on dist-test-slave-z8x9"
server_key: "e857eaed6f47aa89a95c92b0d991ecae"
server_key_iv: "73e5cbde04ab8fe59499c8755f05edcd"
server_key_version: "encryptionkey@0"
I20260807 22:59:52.671927 16905 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260807 22:59:52.675532 16929 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:59:52.676939 16905 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260807 22:59:52.677121 16905 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-1/wal
uuid: "102b2e12e0f546e0a4f6402e8bd91481"
format_stamp: "Formatted at 2026-08-07 22:59:52 on dist-test-slave-z8x9"
server_key: "e857eaed6f47aa89a95c92b0d991ecae"
server_key_iv: "73e5cbde04ab8fe59499c8755f05edcd"
server_key_version: "encryptionkey@0"
I20260807 22:59:52.677389 16905 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:59:52.695990 16905 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:59:52.700433 16905 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:59:52.700758 16905 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:59:52.702018 16905 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:59:52.703940 16905 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:59:52.704043 16905 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:59:52.704200 16905 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:59:52.704278 16905 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:59:52.741669 16905 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.2:39593
I20260807 22:59:52.741681 17127 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.2:39593 every 8 connection(s)
I20260807 22:59:52.743460 16905 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-1/data/info.pb
I20260807 22:59:52.744643 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 16905
I20260807 22:59:52.744855 25976 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-1/wal/instance
I20260807 22:59:52.745484 25976 external_mini_cluster.cc:1428] Setting key c27dc0c7456d80a38376b89af3bbc684
Aug 07 22:59:52 dist-test-slave-z8x9 krb5kdc[16525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143592, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.2@KRBTEST.COM for kudu/127.25.94.62@KRBTEST.COM
I20260807 22:59:52.774068 17134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:52.747048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:42307 (local address 127.25.94.2:47041)
0807 22:59:52.753887 (+ 6839us) negotiation.cc:107] Waiting for socket to connect
0807 22:59:52.753948 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0807 22:59:52.755683 (+ 1735us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:59:52.756468 (+ 785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:59:52.756477 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:59:52.757012 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:59:52.757685 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:59:52.757704 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:59:52.758800 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:59:52.758804 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:59:52.760040 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:59:52.760052 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:59:52.760884 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:59:52.760891 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:59:52.761123 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:59:52.761903 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:59:52.761930 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:59:52.764934 (+ 3004us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:59:52.769361 (+ 4427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:59:52.769370 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:59:52.769387 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:59:52.769961 (+ 574us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:59:52.770415 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:59:52.770420 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:59:52.770424 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:59:52.770658 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:59:52.771315 (+ 657us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:59:52.771323 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:59:52.771737 (+ 414us) client_negotiation.cc:769] Sending connection context
0807 22:59:52.772026 (+ 289us) client_negotiation.cc:240] Negotiation successful
0807 22:59:52.772563 (+ 537us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6528,"thread_start_us":6312,"threads_started":1}
I20260807 22:59:52.774333 17137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:52.747598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.2:47041 (local address 127.25.94.62:42307)
0807 22:59:52.748127 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0807 22:59:52.748133 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 22:59:52.754359 (+ 6226us) server_negotiation.cc:407] Connection header received
0807 22:59:52.755982 (+ 1623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:59:52.755988 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:59:52.756087 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:59:52.756207 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:59:52.757862 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:59:52.758630 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:59:52.760188 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:59:52.760630 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:59:52.760762 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:59:52.765276 (+ 4514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:59:52.765283 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:59:52.765288 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:59:52.765328 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:59:52.769206 (+ 3878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:59:52.770127 (+ 921us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:59:52.770132 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:59:52.770135 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:59:52.770291 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:59:52.770801 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:59:52.770806 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:59:52.770809 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:59:52.771169 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:59:52.773351 (+ 2182us) server_negotiation.cc:1035] Waiting for connection context
0807 22:59:52.773527 (+ 176us) server_negotiation.cc:299] Negotiation successful
0807 22:59:52.773797 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":277,"threads_started":1}
I20260807 22:59:52.776137 17129 heartbeater.cc:347] Connected to a master server at 127.25.94.62:42307
I20260807 22:59:52.776584 17129 heartbeater.cc:464] Registering TS with master...
I20260807 22:59:52.777678 17129 heartbeater.cc:511] Master 127.25.94.62:42307 requested a full tablet report, sending...
I20260807 22:59:52.779709 16600 ts_manager.cc:194] Registered new tserver with Master: 102b2e12e0f546e0a4f6402e8bd91481 (127.25.94.2:39593)
WARNING: no policy specified for kudu/127.25.94.3@KRBTEST.COM; defaulting to no policy
I20260807 22:59:52.781001 16600 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.94.2@KRBTEST.COM'} at 127.25.94.2:47041
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.3.keytab.
Entry for principal kudu/127.25.94.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.3.keytab.
I20260807 22:59:52.824803 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.3:0
--local_ip_for_outbound_sockets=127.25.94.3
--webserver_interface=127.25.94.3
--webserver_port=0
--tserver_master_addrs=127.25.94.62:42307
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.3.keytab
--principal=kudu/127.25.94.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.20:41671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:59:53.173278 17141 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:59:53.173767 17141 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:59:53.173929 17141 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:59:53.181845 17141 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:59:53.181982 17141 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:59:53.182164 17141 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.3
I20260807 22:59:53.193284 17141 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:41671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.3.keytab
--principal=kudu/127.25.94.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.94.3
--webserver_port=0
--tserver_master_addrs=127.25.94.62:42307
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.3
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 22:59:53.194713 17141 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:59:53.196187 17141 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:59:53.211160 17156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:59:53.211342 17147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:59:53.211297 17151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:59:53.212337 17141 server_base.cc:1034] running on GCE node
I20260807 22:59:53.213295 17141 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:59:53.214490 17141 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:59:53.216012 17141 hybrid_clock.cc:648] HybridClock initialized: now 1786143593215919 us; error 67 us; skew 500 ppm
I20260807 22:59:53.216589 17141 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 22:59:53 dist-test-slave-z8x9 krb5kdc[16525](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143593, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:59:53.220963 17141 init.cc:377] Logged in from keytab as kudu/127.25.94.3@KRBTEST.COM (short username kudu)
I20260807 22:59:53.222836 17141 webserver.cc:460] Webserver started at http://127.25.94.3:39035/ using document root <none> and password file <none>
I20260807 22:59:53.223618 17141 fs_manager.cc:359] Metadata directory not provided
I20260807 22:59:53.223789 17141 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:59:53.224113 17141 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:59:53.227257 17141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-2/data/instance:
uuid: "1601d737452f4cb5bd908fbd84c183a9"
format_stamp: "Formatted at 2026-08-07 22:59:53 on dist-test-slave-z8x9"
server_key: "5269a922dcf25da48da9261b5886e089"
server_key_iv: "ff26757ea98e869d4258b576bf912e45"
server_key_version: "encryptionkey@0"
I20260807 22:59:53.228173 17141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-2/wal/instance:
uuid: "1601d737452f4cb5bd908fbd84c183a9"
format_stamp: "Formatted at 2026-08-07 22:59:53 on dist-test-slave-z8x9"
server_key: "5269a922dcf25da48da9261b5886e089"
server_key_iv: "ff26757ea98e869d4258b576bf912e45"
server_key_version: "encryptionkey@0"
I20260807 22:59:53.228297 16897 heartbeater.cc:503] Master 127.25.94.62:42307 was elected leader, sending a full tablet report...
I20260807 22:59:53.234279 17141 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260807 22:59:53.237926 17173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:59:53.239454 17141 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 22:59:53.239666 17141 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-2/wal
uuid: "1601d737452f4cb5bd908fbd84c183a9"
format_stamp: "Formatted at 2026-08-07 22:59:53 on dist-test-slave-z8x9"
server_key: "5269a922dcf25da48da9261b5886e089"
server_key_iv: "ff26757ea98e869d4258b576bf912e45"
server_key_version: "encryptionkey@0"
I20260807 22:59:53.239881 17141 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:59:53.282498 17141 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:59:53.287417 17141 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:59:53.287801 17141 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:59:53.288959 17141 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:59:53.290424 17141 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:59:53.290498 17141 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:59:53.290661 17141 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:59:53.290715 17141 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:59:53.327690 17141 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.3:34735
I20260807 22:59:53.327739 17357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.3:34735 every 8 connection(s)
I20260807 22:59:53.328991 17141 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-2/data/info.pb
I20260807 22:59:53.330778 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 17141
I20260807 22:59:53.330983 25976 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-2/wal/instance
I20260807 22:59:53.331553 25976 external_mini_cluster.cc:1428] Setting key 78438308f6d8778ea7830c3172accaa3
Aug 07 22:59:53 dist-test-slave-z8x9 krb5kdc[16525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143593, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.3@KRBTEST.COM for kudu/127.25.94.62@KRBTEST.COM
I20260807 22:59:53.351861 17362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:53.332026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.3:53059 (local address 127.25.94.62:42307)
0807 22:59:53.332568 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0807 22:59:53.332574 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 22:59:53.333794 (+ 1220us) server_negotiation.cc:407] Connection header received
0807 22:59:53.336331 (+ 2537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:59:53.336336 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:59:53.336437 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:59:53.336560 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:59:53.338342 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:59:53.339138 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:59:53.340734 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:59:53.341147 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:59:53.341357 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:59:53.345341 (+ 3984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:59:53.345347 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:59:53.345350 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:59:53.345394 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:59:53.348560 (+ 3166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:59:53.349461 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:59:53.349465 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:59:53.349468 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:59:53.349594 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:59:53.350120 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:59:53.350124 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:59:53.350127 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:59:53.350499 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:59:53.350627 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0807 22:59:53.351367 (+ 740us) server_negotiation.cc:299] Negotiation successful
0807 22:59:53.351570 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":297,"threads_started":1}
I20260807 22:59:53.352864 17364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:53.332575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:42307 (local address 127.25.94.3:53059)
0807 22:59:53.333613 (+ 1038us) negotiation.cc:107] Waiting for socket to connect
0807 22:59:53.333666 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0807 22:59:53.336030 (+ 2364us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:59:53.336828 (+ 798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:59:53.336838 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:59:53.337504 (+ 666us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:59:53.338138 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:59:53.338156 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:59:53.339309 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:59:53.339316 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:59:53.340572 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:59:53.340589 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:59:53.341453 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:59:53.341461 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:59:53.341707 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:59:53.342411 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:59:53.342436 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:59:53.345034 (+ 2598us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:59:53.348709 (+ 3675us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:59:53.348717 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:59:53.348731 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:59:53.349346 (+ 615us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:59:53.349741 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:59:53.349746 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:59:53.349748 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:59:53.349957 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:59:53.350634 (+ 677us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:59:53.350641 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:59:53.350992 (+ 351us) client_negotiation.cc:769] Sending connection context
0807 22:59:53.351240 (+ 248us) client_negotiation.cc:240] Negotiation successful
0807 22:59:53.351685 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":730,"thread_start_us":478,"threads_started":1}
I20260807 22:59:53.354357 17358 heartbeater.cc:347] Connected to a master server at 127.25.94.62:42307
I20260807 22:59:53.354686 17358 heartbeater.cc:464] Registering TS with master...
I20260807 22:59:53.355458 17358 heartbeater.cc:511] Master 127.25.94.62:42307 requested a full tablet report, sending...
I20260807 22:59:53.356940 16600 ts_manager.cc:194] Registered new tserver with Master: 1601d737452f4cb5bd908fbd84c183a9 (127.25.94.3:34735)
I20260807 22:59:53.357832 16600 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.94.3@KRBTEST.COM'} at 127.25.94.3:53059
I20260807 22:59:53.360947 25976 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 22:59:53.380340 17362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:53.366823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51480 (local address 127.25.94.62:42307)
0807 22:59:53.367089 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0807 22:59:53.367096 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 22:59:53.367375 (+ 279us) server_negotiation.cc:407] Connection header received
0807 22:59:53.367506 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:59:53.367510 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:59:53.367605 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:59:53.367700 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:59:53.369304 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:59:53.370084 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:59:53.371580 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:59:53.372080 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:59:53.372180 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:59:53.374733 (+ 2553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:59:53.374738 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:59:53.374741 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:59:53.374775 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:59:53.377656 (+ 2881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:59:53.378420 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:59:53.378424 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:59:53.378426 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:59:53.378547 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:59:53.378911 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:59:53.378914 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:59:53.378916 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:59:53.379265 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:59:53.379453 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0807 22:59:53.379796 (+ 343us) server_negotiation.cc:299] Negotiation successful
0807 22:59:53.379969 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260807 22:59:53.384614 16600 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51480:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 22:59:53.392117 16600 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 22:59:53.419538 17386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:53.406629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.3:34735 (local address 127.0.0.1:50272)
0807 22:59:53.411069 (+ 4440us) negotiation.cc:107] Waiting for socket to connect
0807 22:59:53.411088 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 22:59:53.411253 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:59:53.411856 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:59:53.411863 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:59:53.411897 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:59:53.412360 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:59:53.412374 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:59:53.414349 (+ 1975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:59:53.414355 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:59:53.416543 (+ 2188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:59:53.416557 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:59:53.418444 (+ 1887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:59:53.418449 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:59:53.418618 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:59:53.418658 (+ 40us) client_negotiation.cc:769] Sending connection context
0807 22:59:53.418708 (+ 50us) client_negotiation.cc:240] Negotiation successful
0807 22:59:53.418955 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4309,"mutex_wait_us":41,"thread_start_us":1034,"threads_started":1}
I20260807 22:59:53.419673 17387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:53.406688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50288 (local address 127.25.94.2:39593)
0807 22:59:53.410181 (+ 3493us) server_negotiation.cc:206] Beginning negotiation
0807 22:59:53.410192 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 22:59:53.410223 (+ 31us) server_negotiation.cc:407] Connection header received
0807 22:59:53.410328 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:59:53.410337 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:59:53.410612 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:59:53.411478 (+ 866us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:59:53.411722 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:59:53.413260 (+ 1538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:59:53.416311 (+ 3051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:59:53.417960 (+ 1649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:59:53.418828 (+ 868us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:59:53.419041 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0807 22:59:53.419118 (+ 77us) server_negotiation.cc:299] Negotiation successful
0807 22:59:53.419296 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3316,"thread_start_us":3696,"threads_started":1}
I20260807 22:59:53.419589 17385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:53.406601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.2:39593 (local address 127.0.0.1:50288)
0807 22:59:53.407438 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0807 22:59:53.407493 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0807 22:59:53.407687 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:59:53.410923 (+ 3236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:59:53.410932 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:59:53.410976 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:59:53.411485 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:59:53.411504 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:59:53.413436 (+ 1932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:59:53.413442 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:59:53.416081 (+ 2639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:59:53.416097 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:59:53.418224 (+ 2127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:59:53.418231 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:59:53.418472 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:59:53.418544 (+ 72us) client_negotiation.cc:769] Sending connection context
0807 22:59:53.418664 (+ 120us) client_negotiation.cc:240] Negotiation successful
0807 22:59:53.418964 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"mutex_wait_us":56,"thread_start_us":4166,"threads_started":1}
I20260807 22:59:53.420866 17389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:53.407404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50272 (local address 127.25.94.3:34735)
0807 22:59:53.409718 (+ 2314us) server_negotiation.cc:206] Beginning negotiation
0807 22:59:53.409726 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 22:59:53.411202 (+ 1476us) server_negotiation.cc:407] Connection header received
0807 22:59:53.411428 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:59:53.411436 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:59:53.411649 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:59:53.411859 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:59:53.412559 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:59:53.414181 (+ 1622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:59:53.416741 (+ 2560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:59:53.418207 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:59:53.420144 (+ 1937us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:59:53.420404 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0807 22:59:53.420460 (+ 56us) server_negotiation.cc:299] Negotiation successful
0807 22:59:53.420581 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2151,"thread_start_us":1995,"threads_started":1}
I20260807 22:59:53.423738 17391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:53.410343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.1:46643 (local address 127.0.0.1:51254)
0807 22:59:53.414377 (+ 4034us) negotiation.cc:107] Waiting for socket to connect
0807 22:59:53.414396 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 22:59:53.414544 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:59:53.415123 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:59:53.415127 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:59:53.415152 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:59:53.415597 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:59:53.415606 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:59:53.416983 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:59:53.416987 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:59:53.418951 (+ 1964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:59:53.418966 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:59:53.422860 (+ 3894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:59:53.422865 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:59:53.423075 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:59:53.423100 (+ 25us) client_negotiation.cc:769] Sending connection context
0807 22:59:53.423208 (+ 108us) client_negotiation.cc:240] Negotiation successful
0807 22:59:53.423364 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3905,"thread_start_us":3744,"threads_started":1}
I20260807 22:59:53.423800 17393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:53.411014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51254 (local address 127.25.94.1:46643)
0807 22:59:53.413847 (+ 2833us) server_negotiation.cc:206] Beginning negotiation
0807 22:59:53.413858 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 22:59:53.414689 (+ 831us) server_negotiation.cc:407] Connection header received
0807 22:59:53.414780 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:59:53.414786 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:59:53.414941 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:59:53.415131 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:59:53.415799 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:59:53.416815 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:59:53.419179 (+ 2364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:59:53.422326 (+ 3147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:59:53.422478 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:59:53.422722 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0807 22:59:53.423255 (+ 533us) server_negotiation.cc:299] Negotiation successful
0807 22:59:53.423435 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2619,"thread_start_us":2460,"threads_started":1}
I20260807 22:59:53.424268 17010 tablet_service.cc:1467] Processing CreateTablet for tablet a6fa30c87ca346ceb4984a370b2ff632 (DEFAULT_TABLE table=test-table [id=87c788974f9c4ae3a8cd7246f81c59a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:59:53.424271 17248 tablet_service.cc:1467] Processing CreateTablet for tablet a6fa30c87ca346ceb4984a370b2ff632 (DEFAULT_TABLE table=test-table [id=87c788974f9c4ae3a8cd7246f81c59a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:59:53.425803 17010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6fa30c87ca346ceb4984a370b2ff632. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:59:53.425796 17248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6fa30c87ca346ceb4984a370b2ff632. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:59:53.426941 16788 tablet_service.cc:1467] Processing CreateTablet for tablet a6fa30c87ca346ceb4984a370b2ff632 (DEFAULT_TABLE table=test-table [id=87c788974f9c4ae3a8cd7246f81c59a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:59:53.428383 16788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6fa30c87ca346ceb4984a370b2ff632. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:59:53.440012 17395 tablet_bootstrap.cc:492] T a6fa30c87ca346ceb4984a370b2ff632 P cd086879c54a40bdb4dfecda24a0736b: Bootstrap starting.
I20260807 22:59:53.445472 17395 tablet_bootstrap.cc:654] T a6fa30c87ca346ceb4984a370b2ff632 P cd086879c54a40bdb4dfecda24a0736b: Neither blocks nor log segments found. Creating new log.
I20260807 22:59:53.446605 17395 log.cc:824] T a6fa30c87ca346ceb4984a370b2ff632 P cd086879c54a40bdb4dfecda24a0736b: Log is configured to *not* fsync() on all Append() calls
I20260807 22:59:53.447376 17396 tablet_bootstrap.cc:492] T a6fa30c87ca346ceb4984a370b2ff632 P 102b2e12e0f546e0a4f6402e8bd91481: Bootstrap starting.
I20260807 22:59:53.447669 17397 tablet_bootstrap.cc:492] T a6fa30c87ca346ceb4984a370b2ff632 P 1601d737452f4cb5bd908fbd84c183a9: Bootstrap starting.
I20260807 22:59:53.450595 17395 tablet_bootstrap.cc:492] T a6fa30c87ca346ceb4984a370b2ff632 P cd086879c54a40bdb4dfecda24a0736b: No bootstrap required, opened a new log
I20260807 22:59:53.450903 17395 ts_tablet_manager.cc:1397] T a6fa30c87ca346ceb4984a370b2ff632 P cd086879c54a40bdb4dfecda24a0736b: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.005s
I20260807 22:59:53.451284 17396 tablet_bootstrap.cc:654] T a6fa30c87ca346ceb4984a370b2ff632 P 102b2e12e0f546e0a4f6402e8bd91481: Neither blocks nor log segments found. Creating new log.
I20260807 22:59:53.451301 17397 tablet_bootstrap.cc:654] T a6fa30c87ca346ceb4984a370b2ff632 P 1601d737452f4cb5bd908fbd84c183a9: Neither blocks nor log segments found. Creating new log.
I20260807 22:59:53.452625 17396 log.cc:824] T a6fa30c87ca346ceb4984a370b2ff632 P 102b2e12e0f546e0a4f6402e8bd91481: Log is configured to *not* fsync() on all Append() calls
I20260807 22:59:53.452623 17397 log.cc:824] T a6fa30c87ca346ceb4984a370b2ff632 P 1601d737452f4cb5bd908fbd84c183a9: Log is configured to *not* fsync() on all Append() calls
I20260807 22:59:53.455758 17396 tablet_bootstrap.cc:492] T a6fa30c87ca346ceb4984a370b2ff632 P 102b2e12e0f546e0a4f6402e8bd91481: No bootstrap required, opened a new log
I20260807 22:59:53.455976 17396 ts_tablet_manager.cc:1397] T a6fa30c87ca346ceb4984a370b2ff632 P 102b2e12e0f546e0a4f6402e8bd91481: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.007s
I20260807 22:59:53.458925 17395 raft_consensus.cc:348] T a6fa30c87ca346ceb4984a370b2ff632 P cd086879c54a40bdb4dfecda24a0736b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "102b2e12e0f546e0a4f6402e8bd91481" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 39593 } } peers { permanent_uuid: "1601d737452f4cb5bd908fbd84c183a9" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 34735 } } peers { permanent_uuid: "cd086879c54a40bdb4dfecda24a0736b" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 46643 } }
I20260807 22:59:53.459357 17395 raft_consensus.cc:374] T a6fa30c87ca346ceb4984a370b2ff632 P cd086879c54a40bdb4dfecda24a0736b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:59:53.459519 17395 raft_consensus.cc:729] T a6fa30c87ca346ceb4984a370b2ff632 P cd086879c54a40bdb4dfecda24a0736b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd086879c54a40bdb4dfecda24a0736b, State: Initialized, Role: FOLLOWER
I20260807 22:59:53.460210 17395 consensus_queue.cc:260] T a6fa30c87ca346ceb4984a370b2ff632 P cd086879c54a40bdb4dfecda24a0736b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "102b2e12e0f546e0a4f6402e8bd91481" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 39593 } } peers { permanent_uuid: "1601d737452f4cb5bd908fbd84c183a9" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 34735 } } peers { permanent_uuid: "cd086879c54a40bdb4dfecda24a0736b" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 46643 } }
I20260807 22:59:53.462302 17395 ts_tablet_manager.cc:1428] T a6fa30c87ca346ceb4984a370b2ff632 P cd086879c54a40bdb4dfecda24a0736b: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260807 22:59:53.463346 17397 tablet_bootstrap.cc:492] T a6fa30c87ca346ceb4984a370b2ff632 P 1601d737452f4cb5bd908fbd84c183a9: No bootstrap required, opened a new log
I20260807 22:59:53.463649 17397 ts_tablet_manager.cc:1397] T a6fa30c87ca346ceb4984a370b2ff632 P 1601d737452f4cb5bd908fbd84c183a9: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.008s
I20260807 22:59:53.464444 17396 raft_consensus.cc:348] T a6fa30c87ca346ceb4984a370b2ff632 P 102b2e12e0f546e0a4f6402e8bd91481 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "102b2e12e0f546e0a4f6402e8bd91481" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 39593 } } peers { permanent_uuid: "1601d737452f4cb5bd908fbd84c183a9" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 34735 } } peers { permanent_uuid: "cd086879c54a40bdb4dfecda24a0736b" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 46643 } }
I20260807 22:59:53.464955 17396 raft_consensus.cc:374] T a6fa30c87ca346ceb4984a370b2ff632 P 102b2e12e0f546e0a4f6402e8bd91481 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:59:53.465080 17396 raft_consensus.cc:729] T a6fa30c87ca346ceb4984a370b2ff632 P 102b2e12e0f546e0a4f6402e8bd91481 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 102b2e12e0f546e0a4f6402e8bd91481, State: Initialized, Role: FOLLOWER
I20260807 22:59:53.465801 17396 consensus_queue.cc:260] T a6fa30c87ca346ceb4984a370b2ff632 P 102b2e12e0f546e0a4f6402e8bd91481 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "102b2e12e0f546e0a4f6402e8bd91481" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 39593 } } peers { permanent_uuid: "1601d737452f4cb5bd908fbd84c183a9" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 34735 } } peers { permanent_uuid: "cd086879c54a40bdb4dfecda24a0736b" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 46643 } }
I20260807 22:59:53.467672 17129 heartbeater.cc:503] Master 127.25.94.62:42307 was elected leader, sending a full tablet report...
I20260807 22:59:53.468493 17396 ts_tablet_manager.cc:1428] T a6fa30c87ca346ceb4984a370b2ff632 P 102b2e12e0f546e0a4f6402e8bd91481: Time spent starting tablet: real 0.012s user 0.008s sys 0.005s
I20260807 22:59:53.473538 17397 raft_consensus.cc:348] T a6fa30c87ca346ceb4984a370b2ff632 P 1601d737452f4cb5bd908fbd84c183a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "102b2e12e0f546e0a4f6402e8bd91481" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 39593 } } peers { permanent_uuid: "1601d737452f4cb5bd908fbd84c183a9" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 34735 } } peers { permanent_uuid: "cd086879c54a40bdb4dfecda24a0736b" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 46643 } }
I20260807 22:59:53.473901 17397 raft_consensus.cc:374] T a6fa30c87ca346ceb4984a370b2ff632 P 1601d737452f4cb5bd908fbd84c183a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:59:53.474037 17397 raft_consensus.cc:729] T a6fa30c87ca346ceb4984a370b2ff632 P 1601d737452f4cb5bd908fbd84c183a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1601d737452f4cb5bd908fbd84c183a9, State: Initialized, Role: FOLLOWER
I20260807 22:59:53.474699 17397 consensus_queue.cc:260] T a6fa30c87ca346ceb4984a370b2ff632 P 1601d737452f4cb5bd908fbd84c183a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "102b2e12e0f546e0a4f6402e8bd91481" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 39593 } } peers { permanent_uuid: "1601d737452f4cb5bd908fbd84c183a9" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 34735 } } peers { permanent_uuid: "cd086879c54a40bdb4dfecda24a0736b" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 46643 } }
I20260807 22:59:53.475778 17358 heartbeater.cc:503] Master 127.25.94.62:42307 was elected leader, sending a full tablet report...
I20260807 22:59:53.476284 17397 ts_tablet_manager.cc:1428] T a6fa30c87ca346ceb4984a370b2ff632 P 1601d737452f4cb5bd908fbd84c183a9: Time spent starting tablet: real 0.013s user 0.010s sys 0.002s
W20260807 22:59:53.495684 17132 tablet.cc:2369] T a6fa30c87ca346ceb4984a370b2ff632 P 102b2e12e0f546e0a4f6402e8bd91481: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 22:59:53.580925 17359 tablet.cc:2369] T a6fa30c87ca346ceb4984a370b2ff632 P 1601d737452f4cb5bd908fbd84c183a9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 22:59:53.692970 16898 tablet.cc:2369] T a6fa30c87ca346ceb4984a370b2ff632 P cd086879c54a40bdb4dfecda24a0736b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 22:59:53.867343 17407 raft_consensus.cc:482] T a6fa30c87ca346ceb4984a370b2ff632 P 1601d737452f4cb5bd908fbd84c183a9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 22:59:53.867687 17407 raft_consensus.cc:504] T a6fa30c87ca346ceb4984a370b2ff632 P 1601d737452f4cb5bd908fbd84c183a9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "102b2e12e0f546e0a4f6402e8bd91481" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 39593 } } peers { permanent_uuid: "1601d737452f4cb5bd908fbd84c183a9" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 34735 } } peers { permanent_uuid: "cd086879c54a40bdb4dfecda24a0736b" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 46643 } }
I20260807 22:59:53.869364 17407 leader_election.cc:288] T a6fa30c87ca346ceb4984a370b2ff632 P 1601d737452f4cb5bd908fbd84c183a9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 102b2e12e0f546e0a4f6402e8bd91481 (127.25.94.2:39593), cd086879c54a40bdb4dfecda24a0736b (127.25.94.1:46643)
I20260807 22:59:53.876673 17387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:53.869916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.3:39397 (local address 127.25.94.2:39593)
0807 22:59:53.870171 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0807 22:59:53.870179 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 22:59:53.870740 (+ 561us) server_negotiation.cc:407] Connection header received
0807 22:59:53.870918 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:59:53.870924 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:59:53.871004 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:59:53.871109 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:59:53.871805 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:59:53.872690 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:59:53.874653 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:59:53.875593 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:59:53.875712 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:59:53.875785 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0807 22:59:53.876249 (+ 464us) server_negotiation.cc:299] Negotiation successful
0807 22:59:53.876397 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260807 22:59:53.876716 17408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:53.869714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.2:39593 (local address 127.25.94.3:39397)
0807 22:59:53.870629 (+ 915us) negotiation.cc:107] Waiting for socket to connect
0807 22:59:53.870645 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 22:59:53.870762 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:59:53.871138 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:59:53.871143 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:59:53.871170 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:59:53.871603 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:59:53.871618 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:59:53.872859 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:59:53.872865 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:59:53.874473 (+ 1608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:59:53.874485 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:59:53.875787 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:59:53.875795 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:59:53.875968 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:59:53.876064 (+ 96us) client_negotiation.cc:769] Sending connection context
0807 22:59:53.876147 (+ 83us) client_negotiation.cc:240] Negotiation successful
0807 22:59:53.876240 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":757,"mutex_wait_us":44,"thread_start_us":591,"threads_started":1}
I20260807 22:59:53.876766 17393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:53.869845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.3:35719 (local address 127.25.94.1:46643)
0807 22:59:53.870110 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0807 22:59:53.870120 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 22:59:53.870949 (+ 829us) server_negotiation.cc:407] Connection header received
0807 22:59:53.871138 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:59:53.871143 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:59:53.871223 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:59:53.871359 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:59:53.872146 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:59:53.872971 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:59:53.874881 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:59:53.875755 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:59:53.875891 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:59:53.875992 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0807 22:59:53.876387 (+ 395us) server_negotiation.cc:299] Negotiation successful
0807 22:59:53.876530 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260807 22:59:53.877347 17409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:53.869830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.1:46643 (local address 127.25.94.3:35719)
0807 22:59:53.870827 (+ 997us) negotiation.cc:107] Waiting for socket to connect
0807 22:59:53.870847 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 22:59:53.870980 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:59:53.871384 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:59:53.871387 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:59:53.871410 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:59:53.871934 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:59:53.871948 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:59:53.873115 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:59:53.873119 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:59:53.874724 (+ 1605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:59:53.874738 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:59:53.875965 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:59:53.875969 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:59:53.876199 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:59:53.876235 (+ 36us) client_negotiation.cc:769] Sending connection context
0807 22:59:53.876330 (+ 95us) client_negotiation.cc:240] Negotiation successful
0807 22:59:53.876448 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":827,"mutex_wait_us":97,"thread_start_us":606,"threads_started":1}
I20260807 22:59:53.877885 17051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6fa30c87ca346ceb4984a370b2ff632" candidate_uuid: "1601d737452f4cb5bd908fbd84c183a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "102b2e12e0f546e0a4f6402e8bd91481" is_pre_election: true
I20260807 22:59:53.878422 17051 raft_consensus.cc:2454] T a6fa30c87ca346ceb4984a370b2ff632 P 102b2e12e0f546e0a4f6402e8bd91481 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1601d737452f4cb5bd908fbd84c183a9 in term 0.
I20260807 22:59:53.878417 16822 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6fa30c87ca346ceb4984a370b2ff632" candidate_uuid: "1601d737452f4cb5bd908fbd84c183a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd086879c54a40bdb4dfecda24a0736b" is_pre_election: true
I20260807 22:59:53.878942 16822 raft_consensus.cc:2454] T a6fa30c87ca346ceb4984a370b2ff632 P cd086879c54a40bdb4dfecda24a0736b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1601d737452f4cb5bd908fbd84c183a9 in term 0.
I20260807 22:59:53.879173 17178 leader_election.cc:302] T a6fa30c87ca346ceb4984a370b2ff632 P 1601d737452f4cb5bd908fbd84c183a9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 102b2e12e0f546e0a4f6402e8bd91481, 1601d737452f4cb5bd908fbd84c183a9; no voters:
I20260807 22:59:53.879559 17407 raft_consensus.cc:2781] T a6fa30c87ca346ceb4984a370b2ff632 P 1601d737452f4cb5bd908fbd84c183a9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 22:59:53.879725 17407 raft_consensus.cc:482] T a6fa30c87ca346ceb4984a370b2ff632 P 1601d737452f4cb5bd908fbd84c183a9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 22:59:53.879844 17407 raft_consensus.cc:3037] T a6fa30c87ca346ceb4984a370b2ff632 P 1601d737452f4cb5bd908fbd84c183a9 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:59:53.882443 17407 raft_consensus.cc:504] T a6fa30c87ca346ceb4984a370b2ff632 P 1601d737452f4cb5bd908fbd84c183a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "102b2e12e0f546e0a4f6402e8bd91481" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 39593 } } peers { permanent_uuid: "1601d737452f4cb5bd908fbd84c183a9" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 34735 } } peers { permanent_uuid: "cd086879c54a40bdb4dfecda24a0736b" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 46643 } }
I20260807 22:59:53.883038 17407 leader_election.cc:288] T a6fa30c87ca346ceb4984a370b2ff632 P 1601d737452f4cb5bd908fbd84c183a9 [CANDIDATE]: Term 1 election: Requested vote from peers 102b2e12e0f546e0a4f6402e8bd91481 (127.25.94.2:39593), cd086879c54a40bdb4dfecda24a0736b (127.25.94.1:46643)
I20260807 22:59:53.883512 16822 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6fa30c87ca346ceb4984a370b2ff632" candidate_uuid: "1601d737452f4cb5bd908fbd84c183a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd086879c54a40bdb4dfecda24a0736b"
I20260807 22:59:53.883538 17051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6fa30c87ca346ceb4984a370b2ff632" candidate_uuid: "1601d737452f4cb5bd908fbd84c183a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "102b2e12e0f546e0a4f6402e8bd91481"
I20260807 22:59:53.883728 16822 raft_consensus.cc:3037] T a6fa30c87ca346ceb4984a370b2ff632 P cd086879c54a40bdb4dfecda24a0736b [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:59:53.883793 17051 raft_consensus.cc:3037] T a6fa30c87ca346ceb4984a370b2ff632 P 102b2e12e0f546e0a4f6402e8bd91481 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:59:53.886026 16822 raft_consensus.cc:2454] T a6fa30c87ca346ceb4984a370b2ff632 P cd086879c54a40bdb4dfecda24a0736b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1601d737452f4cb5bd908fbd84c183a9 in term 1.
I20260807 22:59:53.886456 17179 leader_election.cc:302] T a6fa30c87ca346ceb4984a370b2ff632 P 1601d737452f4cb5bd908fbd84c183a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1601d737452f4cb5bd908fbd84c183a9, cd086879c54a40bdb4dfecda24a0736b; no voters:
I20260807 22:59:53.886754 17407 raft_consensus.cc:2781] T a6fa30c87ca346ceb4984a370b2ff632 P 1601d737452f4cb5bd908fbd84c183a9 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:59:53.887084 17051 raft_consensus.cc:2454] T a6fa30c87ca346ceb4984a370b2ff632 P 102b2e12e0f546e0a4f6402e8bd91481 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1601d737452f4cb5bd908fbd84c183a9 in term 1.
I20260807 22:59:53.887324 17407 raft_consensus.cc:686] T a6fa30c87ca346ceb4984a370b2ff632 P 1601d737452f4cb5bd908fbd84c183a9 [term 1 LEADER]: Becoming Leader. State: Replica: 1601d737452f4cb5bd908fbd84c183a9, State: Running, Role: LEADER
I20260807 22:59:53.888123 17407 consensus_queue.cc:237] T a6fa30c87ca346ceb4984a370b2ff632 P 1601d737452f4cb5bd908fbd84c183a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "102b2e12e0f546e0a4f6402e8bd91481" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 39593 } } peers { permanent_uuid: "1601d737452f4cb5bd908fbd84c183a9" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 34735 } } peers { permanent_uuid: "cd086879c54a40bdb4dfecda24a0736b" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 46643 } }
I20260807 22:59:53.892547 16600 catalog_manager.cc:5495] T a6fa30c87ca346ceb4984a370b2ff632 P 1601d737452f4cb5bd908fbd84c183a9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1601d737452f4cb5bd908fbd84c183a9 (127.25.94.3). New cstate: current_term: 1 leader_uuid: "1601d737452f4cb5bd908fbd84c183a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "102b2e12e0f546e0a4f6402e8bd91481" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 39593 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1601d737452f4cb5bd908fbd84c183a9" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 34735 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd086879c54a40bdb4dfecda24a0736b" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 46643 } health_report { overall_health: UNKNOWN } } }
I20260807 22:59:53.989003 17423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:53.982889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50276 (local address 127.25.94.3:34735)
0807 22:59:53.983360 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0807 22:59:53.983365 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 22:59:53.983383 (+ 18us) server_negotiation.cc:407] Connection header received
0807 22:59:53.983639 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:59:53.983643 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:59:53.983738 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:59:53.983884 (+ 146us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 22:59:53.984285 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:59:53.984981 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:59:53.986600 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:59:53.987009 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:59:53.987116 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:59:53.987696 (+ 580us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 22:59:53.988266 (+ 570us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 22:59:53.988389 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0807 22:59:53.988577 (+ 188us) server_negotiation.cc:299] Negotiation successful
0807 22:59:53.988696 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":245,"threads_started":1}
I20260807 22:59:53.997108 16822 raft_consensus.cc:1261] T a6fa30c87ca346ceb4984a370b2ff632 P cd086879c54a40bdb4dfecda24a0736b [term 1 FOLLOWER]: Refusing update from remote peer 1601d737452f4cb5bd908fbd84c183a9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 22:59:53.997155 17051 raft_consensus.cc:1261] T a6fa30c87ca346ceb4984a370b2ff632 P 102b2e12e0f546e0a4f6402e8bd91481 [term 1 FOLLOWER]: Refusing update from remote peer 1601d737452f4cb5bd908fbd84c183a9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 22:59:53.998225 17412 consensus_queue.cc:1035] T a6fa30c87ca346ceb4984a370b2ff632 P 1601d737452f4cb5bd908fbd84c183a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd086879c54a40bdb4dfecda24a0736b" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 46643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 22:59:53.998610 17407 consensus_queue.cc:1035] T a6fa30c87ca346ceb4984a370b2ff632 P 1601d737452f4cb5bd908fbd84c183a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "102b2e12e0f546e0a4f6402e8bd91481" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 39593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 22:59:54.011574 17425 mvcc.cc:205] Tried to move back new op lower bound from 7316044161001050112 to 7316044160569864192. Current Snapshot: MvccSnapshot[applied={T|T < 7316044161001050112}]
I20260807 22:59:54.013430 17432 mvcc.cc:205] Tried to move back new op lower bound from 7316044161001050112 to 7316044160569864192. Current Snapshot: MvccSnapshot[applied={T|T < 7316044161001050112}]
I20260807 22:59:54.025878 16598 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51480:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 22:59:54.026171 16598 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51480:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 22:59:54.030329 16598 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 70cb141317c44ced96448d499c69dd93: Sending DeleteTablet for 3 replicas of tablet a6fa30c87ca346ceb4984a370b2ff632
I20260807 22:59:54.031620 17010 tablet_service.cc:1514] Processing DeleteTablet for tablet a6fa30c87ca346ceb4984a370b2ff632 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:59:54 UTC) from {username='kudu', principal='kudu/127.25.94.62@KRBTEST.COM'} at 127.0.0.1:50288
I20260807 22:59:54.031689 17248 tablet_service.cc:1514] Processing DeleteTablet for tablet a6fa30c87ca346ceb4984a370b2ff632 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:59:54 UTC) from {username='kudu', principal='kudu/127.25.94.62@KRBTEST.COM'} at 127.0.0.1:50272
I20260807 22:59:54.031739 16788 tablet_service.cc:1514] Processing DeleteTablet for tablet a6fa30c87ca346ceb4984a370b2ff632 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:59:54 UTC) from {username='kudu', principal='kudu/127.25.94.62@KRBTEST.COM'} at 127.0.0.1:51254
I20260807 22:59:54.032521 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 16694
I20260807 22:59:54.041426 17443 tablet_replica.cc:330] T a6fa30c87ca346ceb4984a370b2ff632 P 102b2e12e0f546e0a4f6402e8bd91481: stopping tablet replica
I20260807 22:59:54.041931 17443 raft_consensus.cc:2229] T a6fa30c87ca346ceb4984a370b2ff632 P 102b2e12e0f546e0a4f6402e8bd91481 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 22:59:54.042536 17443 raft_consensus.cc:2258] T a6fa30c87ca346ceb4984a370b2ff632 P 102b2e12e0f546e0a4f6402e8bd91481 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 22:59:54.044713 17443 ts_tablet_manager.cc:1905] T a6fa30c87ca346ceb4984a370b2ff632 P 102b2e12e0f546e0a4f6402e8bd91481: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 22:59:54.048171 17444 tablet_replica.cc:330] T a6fa30c87ca346ceb4984a370b2ff632 P 1601d737452f4cb5bd908fbd84c183a9: stopping tablet replica
I20260807 22:59:54.048691 17444 raft_consensus.cc:2229] T a6fa30c87ca346ceb4984a370b2ff632 P 1601d737452f4cb5bd908fbd84c183a9 [term 1 LEADER]: Raft consensus shutting down.
I20260807 22:59:54.049443 17444 raft_consensus.cc:2258] T a6fa30c87ca346ceb4984a370b2ff632 P 1601d737452f4cb5bd908fbd84c183a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 22:59:54.051529 17444 ts_tablet_manager.cc:1905] T a6fa30c87ca346ceb4984a370b2ff632 P 1601d737452f4cb5bd908fbd84c183a9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 22:59:54.052423 17443 ts_tablet_manager.cc:1918] T a6fa30c87ca346ceb4984a370b2ff632 P 102b2e12e0f546e0a4f6402e8bd91481: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 22:59:54.052605 17443 log.cc:1196] T a6fa30c87ca346ceb4984a370b2ff632 P 102b2e12e0f546e0a4f6402e8bd91481: Deleting WAL directory at /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-1/wal/wals/a6fa30c87ca346ceb4984a370b2ff632
I20260807 22:59:54.052479 17440 tablet_replica.cc:330] T a6fa30c87ca346ceb4984a370b2ff632 P cd086879c54a40bdb4dfecda24a0736b: stopping tablet replica
I20260807 22:59:54.053200 17443 ts_tablet_manager.cc:1939] T a6fa30c87ca346ceb4984a370b2ff632 P 102b2e12e0f546e0a4f6402e8bd91481: Deleting consensus metadata
I20260807 22:59:54.053241 17440 raft_consensus.cc:2229] T a6fa30c87ca346ceb4984a370b2ff632 P cd086879c54a40bdb4dfecda24a0736b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 22:59:54.054164 17440 raft_consensus.cc:2258] T a6fa30c87ca346ceb4984a370b2ff632 P cd086879c54a40bdb4dfecda24a0736b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 22:59:54.055470 16578 catalog_manager.cc:4841] TS 102b2e12e0f546e0a4f6402e8bd91481 (127.25.94.2:39593): tablet a6fa30c87ca346ceb4984a370b2ff632 (table test-table [id=87c788974f9c4ae3a8cd7246f81c59a2]) successfully deleted
I20260807 22:59:54.059872 17440 ts_tablet_manager.cc:1905] T a6fa30c87ca346ceb4984a370b2ff632 P cd086879c54a40bdb4dfecda24a0736b: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 07 22:59:54 dist-test-slave-z8x9 krb5kdc[16525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143590, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.1@KRBTEST.COM
I20260807 22:59:54.061197 17444 ts_tablet_manager.cc:1918] T a6fa30c87ca346ceb4984a370b2ff632 P 1601d737452f4cb5bd908fbd84c183a9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 22:59:54.061401 17444 log.cc:1196] T a6fa30c87ca346ceb4984a370b2ff632 P 1601d737452f4cb5bd908fbd84c183a9: Deleting WAL directory at /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-2/wal/wals/a6fa30c87ca346ceb4984a370b2ff632
I20260807 22:59:54.062022 17444 ts_tablet_manager.cc:1939] T a6fa30c87ca346ceb4984a370b2ff632 P 1601d737452f4cb5bd908fbd84c183a9: Deleting consensus metadata
I20260807 22:59:54.063933 16576 catalog_manager.cc:4841] TS 1601d737452f4cb5bd908fbd84c183a9 (127.25.94.3:34735): tablet a6fa30c87ca346ceb4984a370b2ff632 (table test-table [id=87c788974f9c4ae3a8cd7246f81c59a2]) successfully deleted
I20260807 22:59:54.067473 17440 ts_tablet_manager.cc:1918] T a6fa30c87ca346ceb4984a370b2ff632 P cd086879c54a40bdb4dfecda24a0736b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 22:59:54.067664 17440 log.cc:1196] T a6fa30c87ca346ceb4984a370b2ff632 P cd086879c54a40bdb4dfecda24a0736b: Deleting WAL directory at /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-0/wal/wals/a6fa30c87ca346ceb4984a370b2ff632
I20260807 22:59:54.068136 17440 ts_tablet_manager.cc:1939] T a6fa30c87ca346ceb4984a370b2ff632 P cd086879c54a40bdb4dfecda24a0736b: Deleting consensus metadata
I20260807 22:59:54.070283 16579 catalog_manager.cc:4841] TS cd086879c54a40bdb4dfecda24a0736b (127.25.94.1:46643): tablet a6fa30c87ca346ceb4984a370b2ff632 (table test-table [id=87c788974f9c4ae3a8cd7246f81c59a2]) successfully deleted
I20260807 22:59:54.070885 17393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:54.034109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51258 (local address 127.25.94.1:46643)
0807 22:59:54.034358 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0807 22:59:54.034366 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 22:59:54.041245 (+ 6879us) server_negotiation.cc:407] Connection header received
0807 22:59:54.042918 (+ 1673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:59:54.042925 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:59:54.043059 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:59:54.043160 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:59:54.048566 (+ 5406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:59:54.049738 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:59:54.054688 (+ 4950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:59:54.055378 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:59:54.055494 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:59:54.062107 (+ 6613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:59:54.062129 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:59:54.062148 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:59:54.062205 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:59:54.066741 (+ 4536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:59:54.067871 (+ 1130us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:59:54.067882 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:59:54.067886 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:59:54.068082 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:59:54.068612 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:59:54.068618 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:59:54.068621 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:59:54.069161 (+ 540us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:59:54.069558 (+ 397us) server_negotiation.cc:1035] Waiting for connection context
0807 22:59:54.069982 (+ 424us) server_negotiation.cc:299] Negotiation successful
0807 22:59:54.070222 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260807 22:59:54.071563 16889 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 22:59:54.163439 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 16694
I20260807 22:59:54.188254 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 16905
Aug 07 22:59:54 dist-test-slave-z8x9 krb5kdc[16525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143590, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.2@KRBTEST.COM
I20260807 22:59:54.211696 17387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:54.189113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50298 (local address 127.25.94.2:39593)
0807 22:59:54.189338 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0807 22:59:54.189342 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 22:59:54.189363 (+ 21us) server_negotiation.cc:407] Connection header received
0807 22:59:54.189614 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:59:54.189619 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:59:54.189732 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:59:54.189840 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:59:54.191681 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:59:54.192477 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:59:54.194394 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:59:54.194808 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:59:54.194919 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:59:54.204803 (+ 9884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:59:54.204822 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:59:54.204839 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:59:54.204882 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:59:54.208180 (+ 3298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:59:54.209081 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:59:54.209089 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:59:54.209092 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:59:54.209303 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:59:54.209706 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:59:54.209712 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:59:54.209715 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:59:54.210130 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:59:54.210453 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0807 22:59:54.211027 (+ 574us) server_negotiation.cc:299] Negotiation successful
0807 22:59:54.211311 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260807 22:59:54.212514 17121 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 22:59:54.306054 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 16905
I20260807 22:59:54.337564 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 17141
Aug 07 22:59:54 dist-test-slave-z8x9 krb5kdc[16525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143590, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.3@KRBTEST.COM
I20260807 22:59:54.357722 17423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:54.338443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50280 (local address 127.25.94.3:34735)
0807 22:59:54.338647 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0807 22:59:54.338656 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 22:59:54.338726 (+ 70us) server_negotiation.cc:407] Connection header received
0807 22:59:54.338905 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:59:54.338913 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:59:54.339009 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:59:54.339133 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:59:54.340835 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:59:54.342301 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:59:54.344271 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:59:54.345013 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:59:54.345145 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:59:54.351100 (+ 5955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:59:54.351111 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:59:54.351125 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:59:54.351168 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:59:54.354510 (+ 3342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:59:54.355448 (+ 938us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:59:54.355454 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:59:54.355456 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:59:54.355609 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:59:54.356045 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:59:54.356048 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:59:54.356050 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:59:54.356429 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:59:54.356661 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0807 22:59:54.357092 (+ 431us) server_negotiation.cc:299] Negotiation successful
0807 22:59:54.357373 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260807 22:59:54.358561 17350 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 22:59:54.486711 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 17141
I20260807 22:59:54.518512 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 16546
I20260807 22:59:54.519789 16646 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 22:59:54.630534 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 16546
I20260807 22:59:54.662560 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.62:42307
--webserver_interface=127.25.94.62
--webserver_port=42987
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.62.keytab
--principal=kudu/127.25.94.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.20:41671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.94.62:42307
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:59:55.004560 17451 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:59:55.004889 17451 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:59:55.004982 17451 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:59:55.012732 17451 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 22:59:55.012859 17451 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:59:55.012907 17451 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:59:55.012967 17451 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 22:59:55.013012 17451 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 22:59:55.023249 17451 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:41671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.25.94.62:42307
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.62.keytab
--principal=kudu/127.25.94.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.62:42307
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.94.62
--webserver_port=42987
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 22:59:55.024534 17451 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:59:55.025835 17451 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:59:55.053241 17462 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:59:55.053188 17460 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:59:55.053282 17465 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:59:55.054162 17451 server_base.cc:1034] running on GCE node
I20260807 22:59:55.055159 17451 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:59:55.056809 17451 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:59:55.058054 17451 hybrid_clock.cc:648] HybridClock initialized: now 1786143595058003 us; error 68 us; skew 500 ppm
I20260807 22:59:55.058493 17451 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 22:59:55 dist-test-slave-z8x9 krb5kdc[16525](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143595, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:59:55.062484 17451 init.cc:377] Logged in from keytab as kudu/127.25.94.62@KRBTEST.COM (short username kudu)
I20260807 22:59:55.064419 17451 webserver.cc:460] Webserver started at http://127.25.94.62:42987/ using document root <none> and password file <none>
I20260807 22:59:55.065328 17451 fs_manager.cc:359] Metadata directory not provided
I20260807 22:59:55.065508 17451 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:59:55.071798 17451 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260807 22:59:55.075001 17477 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:59:55.076539 17451 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260807 22:59:55.076757 17451 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/master-0/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/master-0/wal
uuid: "70cb141317c44ced96448d499c69dd93"
format_stamp: "Formatted at 2026-08-07 22:59:51 on dist-test-slave-z8x9"
server_key: "a88c9b3689a8ca5be384dd3b9d50679a"
server_key_iv: "81694a9a1d3615340b77eca7ccef72cd"
server_key_version: "encryptionkey@0"
I20260807 22:59:55.077991 17451 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:59:55.102087 17451 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:59:55.107216 17451 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:59:55.107544 17451 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:59:55.128914 17451 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.62:42307
I20260807 22:59:55.128932 17572 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.62:42307 every 8 connection(s)
I20260807 22:59:55.130656 17451 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
I20260807 22:59:55.133996 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 17451
I20260807 22:59:55.135053 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.1:46643
--local_ip_for_outbound_sockets=127.25.94.1
--tserver_master_addrs=127.25.94.62:42307
--webserver_port=36475
--webserver_interface=127.25.94.1
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.1.keytab
--principal=kudu/127.25.94.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.20:41671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 22:59:55.140550 17573 sys_catalog.cc:263] Verifying existing consensus state
I20260807 22:59:55.146780 17573 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70cb141317c44ced96448d499c69dd93: Bootstrap starting.
I20260807 22:59:55.171072 17573 log.cc:824] T 00000000000000000000000000000000 P 70cb141317c44ced96448d499c69dd93: Log is configured to *not* fsync() on all Append() calls
I20260807 22:59:55.187444 17573 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70cb141317c44ced96448d499c69dd93: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 22:59:55.188112 17573 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70cb141317c44ced96448d499c69dd93: Bootstrap complete.
I20260807 22:59:55.198787 17573 raft_consensus.cc:348] T 00000000000000000000000000000000 P 70cb141317c44ced96448d499c69dd93 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70cb141317c44ced96448d499c69dd93" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 42307 } }
I20260807 22:59:55.199684 17573 raft_consensus.cc:729] T 00000000000000000000000000000000 P 70cb141317c44ced96448d499c69dd93 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70cb141317c44ced96448d499c69dd93, State: Initialized, Role: FOLLOWER
I20260807 22:59:55.200397 17573 consensus_queue.cc:260] T 00000000000000000000000000000000 P 70cb141317c44ced96448d499c69dd93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70cb141317c44ced96448d499c69dd93" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 42307 } }
I20260807 22:59:55.200645 17573 raft_consensus.cc:388] T 00000000000000000000000000000000 P 70cb141317c44ced96448d499c69dd93 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 22:59:55.200770 17573 raft_consensus.cc:482] T 00000000000000000000000000000000 P 70cb141317c44ced96448d499c69dd93 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 22:59:55.200969 17573 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 70cb141317c44ced96448d499c69dd93 [term 1 FOLLOWER]: Advancing to term 2
I20260807 22:59:55.205070 17573 raft_consensus.cc:504] T 00000000000000000000000000000000 P 70cb141317c44ced96448d499c69dd93 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70cb141317c44ced96448d499c69dd93" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 42307 } }
I20260807 22:59:55.205698 17573 leader_election.cc:302] T 00000000000000000000000000000000 P 70cb141317c44ced96448d499c69dd93 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 70cb141317c44ced96448d499c69dd93; no voters:
I20260807 22:59:55.206370 17573 leader_election.cc:288] T 00000000000000000000000000000000 P 70cb141317c44ced96448d499c69dd93 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260807 22:59:55.206513 17578 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 70cb141317c44ced96448d499c69dd93 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 22:59:55.207881 17578 raft_consensus.cc:686] T 00000000000000000000000000000000 P 70cb141317c44ced96448d499c69dd93 [term 2 LEADER]: Becoming Leader. State: Replica: 70cb141317c44ced96448d499c69dd93, State: Running, Role: LEADER
I20260807 22:59:55.208591 17578 consensus_queue.cc:237] T 00000000000000000000000000000000 P 70cb141317c44ced96448d499c69dd93 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70cb141317c44ced96448d499c69dd93" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 42307 } }
I20260807 22:59:55.208834 17573 sys_catalog.cc:564] T 00000000000000000000000000000000 P 70cb141317c44ced96448d499c69dd93 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 22:59:55.211233 17582 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70cb141317c44ced96448d499c69dd93 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 70cb141317c44ced96448d499c69dd93. Latest consensus state: current_term: 2 leader_uuid: "70cb141317c44ced96448d499c69dd93" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70cb141317c44ced96448d499c69dd93" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 42307 } } }
I20260807 22:59:55.211654 17582 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70cb141317c44ced96448d499c69dd93 [sys.catalog]: This master's current role is: LEADER
I20260807 22:59:55.212124 17580 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70cb141317c44ced96448d499c69dd93 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "70cb141317c44ced96448d499c69dd93" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70cb141317c44ced96448d499c69dd93" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 42307 } } }
I20260807 22:59:55.212469 17580 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70cb141317c44ced96448d499c69dd93 [sys.catalog]: This master's current role is: LEADER
I20260807 22:59:55.214726 17590 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 22:59:55.224161 17590 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 22:59:55.226178 17590 catalog_manager.cc:1259] Loaded cluster ID: 3e96fff9e7a54bfabf8a7aca3984e5c7
I20260807 22:59:55.226440 17590 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 22:59:55.231276 17590 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260807 22:59:55.251861 17590 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260807 22:59:55.259733 17590 catalog_manager.cc:1504] Loading token signing keys...
I20260807 22:59:55.261144 17590 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 70cb141317c44ced96448d499c69dd93: Loaded TSK: 0
I20260807 22:59:55.262393 17590 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260807 22:59:55.529379 17576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:59:55.529683 17576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:59:55.529846 17576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:59:55.537302 17576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:59:55.537421 17576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:59:55.537591 17576 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.1
I20260807 22:59:55.547641 17576 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:41671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.1.keytab
--principal=kudu/127.25.94.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.1:46643
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.94.1
--webserver_port=36475
--tserver_master_addrs=127.25.94.62:42307
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.1
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 22:59:55.549033 17576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:59:55.550469 17576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:59:55.564689 17625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:59:55.564971 17621 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:59:55.565609 17620 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:59:55.566264 17576 server_base.cc:1034] running on GCE node
I20260807 22:59:55.566991 17576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:59:55.568109 17576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:59:55.569383 17576 hybrid_clock.cc:648] HybridClock initialized: now 1786143595569321 us; error 55 us; skew 500 ppm
I20260807 22:59:55.569722 17576 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 22:59:55 dist-test-slave-z8x9 krb5kdc[16525](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143595, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:59:55.573315 17576 init.cc:377] Logged in from keytab as kudu/127.25.94.1@KRBTEST.COM (short username kudu)
I20260807 22:59:55.574981 17576 webserver.cc:460] Webserver started at http://127.25.94.1:36475/ using document root <none> and password file <none>
I20260807 22:59:55.575686 17576 fs_manager.cc:359] Metadata directory not provided
I20260807 22:59:55.575860 17576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:59:55.581959 17576 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260807 22:59:55.585273 17636 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:59:55.586899 17576 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 22:59:55.587137 17576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-0/wal
uuid: "cd086879c54a40bdb4dfecda24a0736b"
format_stamp: "Formatted at 2026-08-07 22:59:52 on dist-test-slave-z8x9"
server_key: "bd0710249b124060f98c5a18ebee52b4"
server_key_iv: "b38ebd9c39be82327b298dc096e71404"
server_key_version: "encryptionkey@0"
I20260807 22:59:55.588199 17576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:59:55.603084 17576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:59:55.607514 17576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:59:55.607828 17576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:59:55.608884 17576 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:59:55.610607 17576 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:59:55.610738 17576 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:59:55.610868 17576 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:59:55.610967 17576 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:59:55.645996 17576 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.1:46643
I20260807 22:59:55.646054 17825 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.1:46643 every 8 connection(s)
I20260807 22:59:55.647570 17576 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-0/data/info.pb
I20260807 22:59:55.652258 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 17576
I20260807 22:59:55.652908 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.2:39593
--local_ip_for_outbound_sockets=127.25.94.2
--tserver_master_addrs=127.25.94.62:42307
--webserver_port=42255
--webserver_interface=127.25.94.2
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.2.keytab
--principal=kudu/127.25.94.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.20:41671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 07 22:59:55 dist-test-slave-z8x9 krb5kdc[16525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143595, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.1@KRBTEST.COM for kudu/127.25.94.62@KRBTEST.COM
I20260807 22:59:55.675331 17829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:55.650638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.1:50923 (local address 127.25.94.62:42307)
0807 22:59:55.651280 (+ 642us) server_negotiation.cc:206] Beginning negotiation
0807 22:59:55.651291 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 22:59:55.651760 (+ 469us) server_negotiation.cc:407] Connection header received
0807 22:59:55.654169 (+ 2409us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:59:55.654193 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:59:55.654685 (+ 492us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:59:55.655121 (+ 436us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:59:55.656597 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:59:55.658723 (+ 2126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:59:55.660453 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:59:55.661622 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:59:55.661882 (+ 260us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:59:55.666957 (+ 5075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:59:55.666966 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:59:55.666981 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:59:55.667019 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:59:55.670107 (+ 3088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:59:55.671175 (+ 1068us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:59:55.671182 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:59:55.671188 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:59:55.671377 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:59:55.671912 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:59:55.671915 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:59:55.671917 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:59:55.672432 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:59:55.672659 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0807 22:59:55.674024 (+ 1365us) server_negotiation.cc:299] Negotiation successful
0807 22:59:55.674380 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":304,"threads_started":1}
I20260807 22:59:55.675616 17830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:55.650724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:42307 (local address 127.25.94.1:50923)
0807 22:59:55.651564 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0807 22:59:55.651616 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0807 22:59:55.653521 (+ 1905us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:59:55.655240 (+ 1719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:59:55.655248 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:59:55.655791 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:59:55.656332 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:59:55.656349 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:59:55.658918 (+ 2569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:59:55.658922 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:59:55.660291 (+ 1369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:59:55.660304 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:59:55.661918 (+ 1614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:59:55.661923 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:59:55.662115 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:59:55.662950 (+ 835us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:59:55.662973 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:59:55.666641 (+ 3668us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:59:55.670260 (+ 3619us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:59:55.670272 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:59:55.670290 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:59:55.671005 (+ 715us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:59:55.671529 (+ 524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:59:55.671537 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:59:55.671543 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:59:55.671793 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:59:55.672640 (+ 847us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:59:55.672651 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:59:55.673126 (+ 475us) client_negotiation.cc:769] Sending connection context
0807 22:59:55.673529 (+ 403us) client_negotiation.cc:240] Negotiation successful
0807 22:59:55.674142 (+ 613us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":358,"threads_started":1}
I20260807 22:59:55.678721 17827 heartbeater.cc:347] Connected to a master server at 127.25.94.62:42307
I20260807 22:59:55.679052 17827 heartbeater.cc:464] Registering TS with master...
I20260807 22:59:55.679854 17827 heartbeater.cc:511] Master 127.25.94.62:42307 requested a full tablet report, sending...
I20260807 22:59:55.682210 17510 ts_manager.cc:194] Registered new tserver with Master: cd086879c54a40bdb4dfecda24a0736b (127.25.94.1:46643)
I20260807 22:59:55.683722 17510 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.94.1@KRBTEST.COM'} at 127.25.94.1:50923
W20260807 22:59:55.999361 17831 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:59:55.999707 17831 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:59:55.999890 17831 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:59:56.007778 17831 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:59:56.007972 17831 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:59:56.008152 17831 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.2
I20260807 22:59:56.018541 17831 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:41671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.2.keytab
--principal=kudu/127.25.94.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.2:39593
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.94.2
--webserver_port=42255
--tserver_master_addrs=127.25.94.62:42307
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.2
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 22:59:56.019932 17831 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:59:56.021361 17831 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:59:56.034695 17846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:59:56.035116 17844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:59:56.035317 17848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:59:56.036535 17831 server_base.cc:1034] running on GCE node
I20260807 22:59:56.037125 17831 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:59:56.038125 17831 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:59:56.039332 17831 hybrid_clock.cc:648] HybridClock initialized: now 1786143596039270 us; error 54 us; skew 500 ppm
I20260807 22:59:56.039614 17831 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 22:59:56 dist-test-slave-z8x9 krb5kdc[16525](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143596, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:59:56.043109 17831 init.cc:377] Logged in from keytab as kudu/127.25.94.2@KRBTEST.COM (short username kudu)
I20260807 22:59:56.044675 17831 webserver.cc:460] Webserver started at http://127.25.94.2:42255/ using document root <none> and password file <none>
I20260807 22:59:56.045498 17831 fs_manager.cc:359] Metadata directory not provided
I20260807 22:59:56.045619 17831 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:59:56.051079 17831 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260807 22:59:56.053994 17856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:59:56.055594 17831 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 22:59:56.055796 17831 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-1/wal
uuid: "102b2e12e0f546e0a4f6402e8bd91481"
format_stamp: "Formatted at 2026-08-07 22:59:52 on dist-test-slave-z8x9"
server_key: "e857eaed6f47aa89a95c92b0d991ecae"
server_key_iv: "73e5cbde04ab8fe59499c8755f05edcd"
server_key_version: "encryptionkey@0"
I20260807 22:59:56.056816 17831 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:59:56.100481 17831 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:59:56.105417 17831 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:59:56.105744 17831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:59:56.106837 17831 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:59:56.108268 17831 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:59:56.108386 17831 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:59:56.108529 17831 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:59:56.108625 17831 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:59:56.146085 17831 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.2:39593
I20260807 22:59:56.146142 18041 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.2:39593 every 8 connection(s)
I20260807 22:59:56.147748 17831 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-1/data/info.pb
I20260807 22:59:56.157591 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 17831
I20260807 22:59:56.158408 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.3:34735
--local_ip_for_outbound_sockets=127.25.94.3
--tserver_master_addrs=127.25.94.62:42307
--webserver_port=39035
--webserver_interface=127.25.94.3
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.3.keytab
--principal=kudu/127.25.94.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.20:41671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 07 22:59:56 dist-test-slave-z8x9 krb5kdc[16525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143596, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.2@KRBTEST.COM for kudu/127.25.94.62@KRBTEST.COM
I20260807 22:59:56.176375 17829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:56.150862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.2:49943 (local address 127.25.94.62:42307)
0807 22:59:56.151104 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0807 22:59:56.151110 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 22:59:56.152443 (+ 1333us) server_negotiation.cc:407] Connection header received
0807 22:59:56.154815 (+ 2372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:59:56.154823 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:59:56.155088 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:59:56.155306 (+ 218us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:59:56.157321 (+ 2015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:59:56.158517 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:59:56.161586 (+ 3069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:59:56.162101 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:59:56.162266 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:59:56.168119 (+ 5853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:59:56.168125 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:59:56.168129 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:59:56.168171 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:59:56.171984 (+ 3813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:59:56.172955 (+ 971us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:59:56.172965 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:59:56.172971 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:59:56.173173 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:59:56.173766 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:59:56.173774 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:59:56.173781 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:59:56.174322 (+ 541us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:59:56.174747 (+ 425us) server_negotiation.cc:1035] Waiting for connection context
0807 22:59:56.175578 (+ 831us) server_negotiation.cc:299] Negotiation successful
0807 22:59:56.175912 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260807 22:59:56.177589 18044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:56.151438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:42307 (local address 127.25.94.2:49943)
0807 22:59:56.152253 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0807 22:59:56.152304 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0807 22:59:56.154251 (+ 1947us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:59:56.155713 (+ 1462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:59:56.155723 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:59:56.156323 (+ 600us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:59:56.156989 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:59:56.157009 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:59:56.158774 (+ 1765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:59:56.158783 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:59:56.161289 (+ 2506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:59:56.161314 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:59:56.162408 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:59:56.162418 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:59:56.162794 (+ 376us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:59:56.164088 (+ 1294us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:59:56.164121 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:59:56.167798 (+ 3677us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:59:56.172199 (+ 4401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:59:56.172208 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:59:56.172227 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:59:56.172768 (+ 541us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:59:56.173370 (+ 602us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:59:56.173375 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:59:56.173377 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:59:56.173628 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:59:56.174621 (+ 993us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:59:56.174628 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:59:56.175017 (+ 389us) client_negotiation.cc:769] Sending connection context
0807 22:59:56.175426 (+ 409us) client_negotiation.cc:240] Negotiation successful
0807 22:59:56.176013 (+ 587us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":319,"threads_started":1}
I20260807 22:59:56.179563 18042 heartbeater.cc:347] Connected to a master server at 127.25.94.62:42307
I20260807 22:59:56.179963 18042 heartbeater.cc:464] Registering TS with master...
I20260807 22:59:56.181023 18042 heartbeater.cc:511] Master 127.25.94.62:42307 requested a full tablet report, sending...
I20260807 22:59:56.183255 17510 ts_manager.cc:194] Registered new tserver with Master: 102b2e12e0f546e0a4f6402e8bd91481 (127.25.94.2:39593)
I20260807 22:59:56.184319 17510 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.94.2@KRBTEST.COM'} at 127.25.94.2:49943
W20260807 22:59:56.514734 18045 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:59:56.515017 18045 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:59:56.515221 18045 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:59:56.522514 18045 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:59:56.522622 18045 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:59:56.522825 18045 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.3
I20260807 22:59:56.532526 18045 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:41671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.3.keytab
--principal=kudu/127.25.94.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.3:34735
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.94.3
--webserver_port=39035
--tserver_master_addrs=127.25.94.62:42307
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.3
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 22:59:56.533921 18045 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:59:56.535249 18045 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:59:56.549041 18052 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:59:56.548975 18053 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:59:56.549062 18056 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:59:56.550136 18045 server_base.cc:1034] running on GCE node
I20260807 22:59:56.550781 18045 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:59:56.551839 18045 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:59:56.553042 18045 hybrid_clock.cc:648] HybridClock initialized: now 1786143596552994 us; error 41 us; skew 500 ppm
I20260807 22:59:56.553454 18045 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 22:59:56 dist-test-slave-z8x9 krb5kdc[16525](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143596, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:59:56.556756 18045 init.cc:377] Logged in from keytab as kudu/127.25.94.3@KRBTEST.COM (short username kudu)
I20260807 22:59:56.558267 18045 webserver.cc:460] Webserver started at http://127.25.94.3:39035/ using document root <none> and password file <none>
I20260807 22:59:56.558909 18045 fs_manager.cc:359] Metadata directory not provided
I20260807 22:59:56.559028 18045 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:59:56.564975 18045 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260807 22:59:56.567893 18065 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:59:56.569453 18045 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 22:59:56.569639 18045 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-2/wal
uuid: "1601d737452f4cb5bd908fbd84c183a9"
format_stamp: "Formatted at 2026-08-07 22:59:53 on dist-test-slave-z8x9"
server_key: "5269a922dcf25da48da9261b5886e089"
server_key_iv: "ff26757ea98e869d4258b576bf912e45"
server_key_version: "encryptionkey@0"
I20260807 22:59:56.570655 18045 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:59:56.591549 18045 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:59:56.596022 18045 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:59:56.596336 18045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:59:56.597545 18045 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:59:56.598989 18045 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:59:56.599134 18045 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:59:56.599269 18045 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:59:56.599361 18045 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:59:56.634546 18045 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.3:34735
I20260807 22:59:56.634636 18254 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.3:34735 every 8 connection(s)
I20260807 22:59:56.636013 18045 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-2/data/info.pb
I20260807 22:59:56.636329 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 18045
I20260807 22:59:56.653821 17829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:56.637988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51492 (local address 127.25.94.62:42307)
0807 22:59:56.638198 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0807 22:59:56.638205 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 22:59:56.638228 (+ 23us) server_negotiation.cc:407] Connection header received
0807 22:59:56.638426 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:59:56.638436 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:59:56.638505 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:59:56.638631 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:59:56.640674 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:59:56.641495 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:59:56.643425 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:59:56.643832 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:59:56.643924 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:59:56.647328 (+ 3404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:59:56.647336 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:59:56.647340 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:59:56.647390 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:59:56.650732 (+ 3342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:59:56.651675 (+ 943us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:59:56.651682 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:59:56.651687 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:59:56.651863 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:59:56.652342 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:59:56.652346 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:59:56.652348 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:59:56.652617 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:59:56.652825 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0807 22:59:56.653256 (+ 431us) server_negotiation.cc:299] Negotiation successful
0807 22:59:56.653418 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
Aug 07 22:59:56 dist-test-slave-z8x9 krb5kdc[16525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143596, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.3@KRBTEST.COM for kudu/127.25.94.62@KRBTEST.COM
I20260807 22:59:56.661616 18260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:56.638981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.3:39293 (local address 127.25.94.62:42307)
0807 22:59:56.639474 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0807 22:59:56.639480 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 22:59:56.640669 (+ 1189us) server_negotiation.cc:407] Connection header received
0807 22:59:56.643041 (+ 2372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:59:56.643045 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:59:56.643107 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:59:56.643193 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:59:56.645246 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:59:56.646336 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:59:56.648307 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:59:56.649060 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:59:56.649208 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:59:56.655114 (+ 5906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:59:56.655120 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:59:56.655125 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:59:56.655159 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:59:56.658261 (+ 3102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:59:56.658933 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:59:56.658937 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:59:56.658939 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:59:56.659050 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:59:56.659413 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:59:56.659416 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:59:56.659418 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:59:56.659696 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:59:56.659844 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0807 22:59:56.660766 (+ 922us) server_negotiation.cc:299] Negotiation successful
0807 22:59:56.661068 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":251,"threads_started":1}
I20260807 22:59:56.662812 18261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:56.639587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:42307 (local address 127.25.94.3:39293)
0807 22:59:56.640462 (+ 875us) negotiation.cc:107] Waiting for socket to connect
0807 22:59:56.640513 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0807 22:59:56.642758 (+ 2245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:59:56.643509 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:59:56.643517 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:59:56.644157 (+ 640us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:59:56.644997 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:59:56.645028 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:59:56.646509 (+ 1481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:59:56.646517 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:59:56.648139 (+ 1622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:59:56.648155 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:59:56.650953 (+ 2798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:59:56.650965 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:59:56.651279 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:59:56.652030 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:59:56.652072 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:59:56.654890 (+ 2818us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:59:56.658394 (+ 3504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:59:56.658401 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:59:56.658412 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:59:56.658821 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:59:56.659145 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:59:56.659148 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:59:56.659150 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:59:56.659317 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:59:56.659857 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:59:56.659867 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:59:56.660304 (+ 437us) client_negotiation.cc:769] Sending connection context
0807 22:59:56.660656 (+ 352us) client_negotiation.cc:240] Negotiation successful
0807 22:59:56.661303 (+ 647us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":325,"threads_started":1}
I20260807 22:59:56.664347 18255 heartbeater.cc:347] Connected to a master server at 127.25.94.62:42307
I20260807 22:59:56.664687 18255 heartbeater.cc:464] Registering TS with master...
I20260807 22:59:56.665658 18255 heartbeater.cc:511] Master 127.25.94.62:42307 requested a full tablet report, sending...
I20260807 22:59:56.667452 17510 ts_manager.cc:194] Registered new tserver with Master: 1601d737452f4cb5bd908fbd84c183a9 (127.25.94.3:34735)
I20260807 22:59:56.668697 17510 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.94.3@KRBTEST.COM'} at 127.25.94.3:39293
I20260807 22:59:56.669584 25976 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 22:59:56.676767 18260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:56.670767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51502 (local address 127.25.94.62:42307)
0807 22:59:56.671197 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0807 22:59:56.671203 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 22:59:56.671257 (+ 54us) server_negotiation.cc:407] Connection header received
0807 22:59:56.671523 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:59:56.671528 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:59:56.671674 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:59:56.671826 (+ 152us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 22:59:56.672283 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:59:56.672986 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:59:56.674583 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:59:56.675010 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:59:56.675158 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:59:56.675732 (+ 574us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 22:59:56.676181 (+ 449us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 22:59:56.676336 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0807 22:59:56.676437 (+ 101us) server_negotiation.cc:299] Negotiation successful
0807 22:59:56.676517 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311}
I20260807 22:59:56.677752 17510 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:51502:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 22:59:56.685308 17510 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 22:59:56.686827 17827 heartbeater.cc:503] Master 127.25.94.62:42307 was elected leader, sending a full tablet report...
I20260807 22:59:56.721741 18273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:56.697442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51270 (local address 127.25.94.1:46643)
0807 22:59:56.709997 (+ 12555us) server_negotiation.cc:206] Beginning negotiation
0807 22:59:56.710009 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 22:59:56.711133 (+ 1124us) server_negotiation.cc:407] Connection header received
0807 22:59:56.712051 (+ 918us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:59:56.712064 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:59:56.712331 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:59:56.712540 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:59:56.713757 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:59:56.715768 (+ 2011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:59:56.718296 (+ 2528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:59:56.719825 (+ 1529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:59:56.719957 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:59:56.720195 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0807 22:59:56.721153 (+ 958us) server_negotiation.cc:299] Negotiation successful
0807 22:59:56.721397 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12318,"thread_start_us":12572,"threads_started":1}
I20260807 22:59:56.721757 18271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:56.697616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.2:39593 (local address 127.0.0.1:50312)
0807 22:59:56.711563 (+ 13947us) negotiation.cc:107] Waiting for socket to connect
0807 22:59:56.711582 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 22:59:56.711708 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:59:56.713582 (+ 1874us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:59:56.713590 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:59:56.713626 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:59:56.713998 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:59:56.714008 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:59:56.715921 (+ 1913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:59:56.715926 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:59:56.718312 (+ 2386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:59:56.718329 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:59:56.720390 (+ 2061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:59:56.720397 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:59:56.720768 (+ 371us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:59:56.720865 (+ 97us) client_negotiation.cc:769] Sending connection context
0807 22:59:56.721000 (+ 135us) client_negotiation.cc:240] Negotiation successful
0807 22:59:56.721230 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13777,"mutex_wait_us":185,"thread_start_us":15491,"threads_started":1}
I20260807 22:59:56.721982 18280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:56.698931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50292 (local address 127.25.94.3:34735)
0807 22:59:56.712491 (+ 13560us) server_negotiation.cc:206] Beginning negotiation
0807 22:59:56.712501 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 22:59:56.713809 (+ 1308us) server_negotiation.cc:407] Connection header received
0807 22:59:56.713916 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:59:56.713925 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:59:56.714143 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:59:56.714364 (+ 221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:59:56.715314 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:59:56.716908 (+ 1594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:59:56.719231 (+ 2323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:59:56.720719 (+ 1488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:59:56.720883 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:59:56.721125 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0807 22:59:56.721380 (+ 255us) server_negotiation.cc:299] Negotiation successful
0807 22:59:56.721607 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13360,"thread_start_us":13202,"threads_started":1}
I20260807 22:59:56.722231 18275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:56.697953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.3:34735 (local address 127.0.0.1:50292)
0807 22:59:56.713552 (+ 15599us) negotiation.cc:107] Waiting for socket to connect
0807 22:59:56.713575 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0807 22:59:56.713699 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:59:56.714643 (+ 944us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:59:56.714648 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:59:56.714673 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:59:56.715081 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:59:56.715093 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:59:56.717065 (+ 1972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:59:56.717071 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:59:56.719072 (+ 2001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:59:56.719083 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:59:56.720960 (+ 1877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:59:56.720967 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:59:56.721152 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:59:56.721215 (+ 63us) client_negotiation.cc:769] Sending connection context
0807 22:59:56.721332 (+ 117us) client_negotiation.cc:240] Negotiation successful
0807 22:59:56.721477 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15364,"mutex_wait_us":74,"thread_start_us":12412,"threads_started":1}
I20260807 22:59:56.722604 18276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:56.697249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.1:46643 (local address 127.0.0.1:51270)
0807 22:59:56.710320 (+ 13071us) negotiation.cc:107] Waiting for socket to connect
0807 22:59:56.710999 (+ 679us) client_negotiation.cc:174] Beginning negotiation
0807 22:59:56.711873 (+ 874us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:59:56.712867 (+ 994us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:59:56.712874 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:59:56.712907 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:59:56.713507 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:59:56.713524 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:59:56.715954 (+ 2430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:59:56.715960 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:59:56.718124 (+ 2164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:59:56.718137 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:59:56.720400 (+ 2263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:59:56.720405 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:59:56.720772 (+ 367us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:59:56.720865 (+ 93us) client_negotiation.cc:769] Sending connection context
0807 22:59:56.720993 (+ 128us) client_negotiation.cc:240] Negotiation successful
0807 22:59:56.722074 (+ 1081us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12776,"mutex_wait_us":1094,"thread_start_us":13950,"threads_started":1}
I20260807 22:59:56.722929 18279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:56.698345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50312 (local address 127.25.94.2:39593)
0807 22:59:56.712916 (+ 14571us) server_negotiation.cc:206] Beginning negotiation
0807 22:59:56.712924 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 22:59:56.712952 (+ 28us) server_negotiation.cc:407] Connection header received
0807 22:59:56.713050 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:59:56.713058 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:59:56.713324 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:59:56.713569 (+ 245us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:59:56.714213 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:59:56.715759 (+ 1546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:59:56.718473 (+ 2714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:59:56.719846 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:59:56.719957 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:59:56.720159 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0807 22:59:56.722285 (+ 2126us) server_negotiation.cc:299] Negotiation successful
0807 22:59:56.722546 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14406,"thread_start_us":14263,"threads_started":1}
I20260807 22:59:56.726114 17935 tablet_service.cc:1467] Processing CreateTablet for tablet 8b1b17c6705244ecbf38ffd3c5d6d839 (DEFAULT_TABLE table=test-table [id=e1fb3c2a02804b96b5ffaa177144e4c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:59:56.726204 18146 tablet_service.cc:1467] Processing CreateTablet for tablet 8b1b17c6705244ecbf38ffd3c5d6d839 (DEFAULT_TABLE table=test-table [id=e1fb3c2a02804b96b5ffaa177144e4c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:59:56.726195 17720 tablet_service.cc:1467] Processing CreateTablet for tablet 8b1b17c6705244ecbf38ffd3c5d6d839 (DEFAULT_TABLE table=test-table [id=e1fb3c2a02804b96b5ffaa177144e4c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:59:56.727424 17935 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b1b17c6705244ecbf38ffd3c5d6d839. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:59:56.727592 17720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b1b17c6705244ecbf38ffd3c5d6d839. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:59:56.727584 18146 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b1b17c6705244ecbf38ffd3c5d6d839. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:59:56.740113 18281 tablet_bootstrap.cc:492] T 8b1b17c6705244ecbf38ffd3c5d6d839 P 102b2e12e0f546e0a4f6402e8bd91481: Bootstrap starting.
I20260807 22:59:56.744174 18281 tablet_bootstrap.cc:654] T 8b1b17c6705244ecbf38ffd3c5d6d839 P 102b2e12e0f546e0a4f6402e8bd91481: Neither blocks nor log segments found. Creating new log.
I20260807 22:59:56.745633 18281 log.cc:824] T 8b1b17c6705244ecbf38ffd3c5d6d839 P 102b2e12e0f546e0a4f6402e8bd91481: Log is configured to *not* fsync() on all Append() calls
I20260807 22:59:56.749291 18281 tablet_bootstrap.cc:492] T 8b1b17c6705244ecbf38ffd3c5d6d839 P 102b2e12e0f546e0a4f6402e8bd91481: No bootstrap required, opened a new log
I20260807 22:59:56.749584 18281 ts_tablet_manager.cc:1397] T 8b1b17c6705244ecbf38ffd3c5d6d839 P 102b2e12e0f546e0a4f6402e8bd91481: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260807 22:59:56.750038 18283 tablet_bootstrap.cc:492] T 8b1b17c6705244ecbf38ffd3c5d6d839 P 1601d737452f4cb5bd908fbd84c183a9: Bootstrap starting.
I20260807 22:59:56.753134 18283 tablet_bootstrap.cc:654] T 8b1b17c6705244ecbf38ffd3c5d6d839 P 1601d737452f4cb5bd908fbd84c183a9: Neither blocks nor log segments found. Creating new log.
I20260807 22:59:56.753464 18282 tablet_bootstrap.cc:492] T 8b1b17c6705244ecbf38ffd3c5d6d839 P cd086879c54a40bdb4dfecda24a0736b: Bootstrap starting.
I20260807 22:59:56.754598 18283 log.cc:824] T 8b1b17c6705244ecbf38ffd3c5d6d839 P 1601d737452f4cb5bd908fbd84c183a9: Log is configured to *not* fsync() on all Append() calls
I20260807 22:59:56.757365 18282 tablet_bootstrap.cc:654] T 8b1b17c6705244ecbf38ffd3c5d6d839 P cd086879c54a40bdb4dfecda24a0736b: Neither blocks nor log segments found. Creating new log.
I20260807 22:59:56.758328 18283 tablet_bootstrap.cc:492] T 8b1b17c6705244ecbf38ffd3c5d6d839 P 1601d737452f4cb5bd908fbd84c183a9: No bootstrap required, opened a new log
I20260807 22:59:56.758637 18283 ts_tablet_manager.cc:1397] T 8b1b17c6705244ecbf38ffd3c5d6d839 P 1601d737452f4cb5bd908fbd84c183a9: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260807 22:59:56.758674 18282 log.cc:824] T 8b1b17c6705244ecbf38ffd3c5d6d839 P cd086879c54a40bdb4dfecda24a0736b: Log is configured to *not* fsync() on all Append() calls
I20260807 22:59:56.758984 18281 raft_consensus.cc:348] T 8b1b17c6705244ecbf38ffd3c5d6d839 P 102b2e12e0f546e0a4f6402e8bd91481 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd086879c54a40bdb4dfecda24a0736b" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 46643 } } peers { permanent_uuid: "102b2e12e0f546e0a4f6402e8bd91481" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 39593 } } peers { permanent_uuid: "1601d737452f4cb5bd908fbd84c183a9" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 34735 } }
I20260807 22:59:56.759490 18281 raft_consensus.cc:374] T 8b1b17c6705244ecbf38ffd3c5d6d839 P 102b2e12e0f546e0a4f6402e8bd91481 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:59:56.759601 18281 raft_consensus.cc:729] T 8b1b17c6705244ecbf38ffd3c5d6d839 P 102b2e12e0f546e0a4f6402e8bd91481 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 102b2e12e0f546e0a4f6402e8bd91481, State: Initialized, Role: FOLLOWER
I20260807 22:59:56.760432 18281 consensus_queue.cc:260] T 8b1b17c6705244ecbf38ffd3c5d6d839 P 102b2e12e0f546e0a4f6402e8bd91481 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd086879c54a40bdb4dfecda24a0736b" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 46643 } } peers { permanent_uuid: "102b2e12e0f546e0a4f6402e8bd91481" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 39593 } } peers { permanent_uuid: "1601d737452f4cb5bd908fbd84c183a9" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 34735 } }
I20260807 22:59:56.761673 18042 heartbeater.cc:503] Master 127.25.94.62:42307 was elected leader, sending a full tablet report...
I20260807 22:59:56.762326 18281 ts_tablet_manager.cc:1428] T 8b1b17c6705244ecbf38ffd3c5d6d839 P 102b2e12e0f546e0a4f6402e8bd91481: Time spent starting tablet: real 0.013s user 0.011s sys 0.000s
I20260807 22:59:56.762585 18282 tablet_bootstrap.cc:492] T 8b1b17c6705244ecbf38ffd3c5d6d839 P cd086879c54a40bdb4dfecda24a0736b: No bootstrap required, opened a new log
I20260807 22:59:56.762913 18282 ts_tablet_manager.cc:1397] T 8b1b17c6705244ecbf38ffd3c5d6d839 P cd086879c54a40bdb4dfecda24a0736b: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.006s
I20260807 22:59:56.767735 18283 raft_consensus.cc:348] T 8b1b17c6705244ecbf38ffd3c5d6d839 P 1601d737452f4cb5bd908fbd84c183a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd086879c54a40bdb4dfecda24a0736b" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 46643 } } peers { permanent_uuid: "102b2e12e0f546e0a4f6402e8bd91481" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 39593 } } peers { permanent_uuid: "1601d737452f4cb5bd908fbd84c183a9" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 34735 } }
I20260807 22:59:56.768112 18283 raft_consensus.cc:374] T 8b1b17c6705244ecbf38ffd3c5d6d839 P 1601d737452f4cb5bd908fbd84c183a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:59:56.768239 18283 raft_consensus.cc:729] T 8b1b17c6705244ecbf38ffd3c5d6d839 P 1601d737452f4cb5bd908fbd84c183a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1601d737452f4cb5bd908fbd84c183a9, State: Initialized, Role: FOLLOWER
I20260807 22:59:56.768887 18283 consensus_queue.cc:260] T 8b1b17c6705244ecbf38ffd3c5d6d839 P 1601d737452f4cb5bd908fbd84c183a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd086879c54a40bdb4dfecda24a0736b" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 46643 } } peers { permanent_uuid: "102b2e12e0f546e0a4f6402e8bd91481" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 39593 } } peers { permanent_uuid: "1601d737452f4cb5bd908fbd84c183a9" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 34735 } }
I20260807 22:59:56.770346 18255 heartbeater.cc:503] Master 127.25.94.62:42307 was elected leader, sending a full tablet report...
I20260807 22:59:56.771086 18283 ts_tablet_manager.cc:1428] T 8b1b17c6705244ecbf38ffd3c5d6d839 P 1601d737452f4cb5bd908fbd84c183a9: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260807 22:59:56.773087 18282 raft_consensus.cc:348] T 8b1b17c6705244ecbf38ffd3c5d6d839 P cd086879c54a40bdb4dfecda24a0736b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd086879c54a40bdb4dfecda24a0736b" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 46643 } } peers { permanent_uuid: "102b2e12e0f546e0a4f6402e8bd91481" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 39593 } } peers { permanent_uuid: "1601d737452f4cb5bd908fbd84c183a9" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 34735 } }
I20260807 22:59:56.773600 18282 raft_consensus.cc:374] T 8b1b17c6705244ecbf38ffd3c5d6d839 P cd086879c54a40bdb4dfecda24a0736b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:59:56.773733 18282 raft_consensus.cc:729] T 8b1b17c6705244ecbf38ffd3c5d6d839 P cd086879c54a40bdb4dfecda24a0736b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd086879c54a40bdb4dfecda24a0736b, State: Initialized, Role: FOLLOWER
I20260807 22:59:56.774395 18282 consensus_queue.cc:260] T 8b1b17c6705244ecbf38ffd3c5d6d839 P cd086879c54a40bdb4dfecda24a0736b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd086879c54a40bdb4dfecda24a0736b" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 46643 } } peers { permanent_uuid: "102b2e12e0f546e0a4f6402e8bd91481" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 39593 } } peers { permanent_uuid: "1601d737452f4cb5bd908fbd84c183a9" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 34735 } }
I20260807 22:59:56.776062 18282 ts_tablet_manager.cc:1428] T 8b1b17c6705244ecbf38ffd3c5d6d839 P cd086879c54a40bdb4dfecda24a0736b: Time spent starting tablet: real 0.013s user 0.011s sys 0.003s
W20260807 22:59:56.887919 18256 tablet.cc:2369] T 8b1b17c6705244ecbf38ffd3c5d6d839 P 1601d737452f4cb5bd908fbd84c183a9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 22:59:56.900137 17828 tablet.cc:2369] T 8b1b17c6705244ecbf38ffd3c5d6d839 P cd086879c54a40bdb4dfecda24a0736b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 22:59:56.900545 18043 tablet.cc:2369] T 8b1b17c6705244ecbf38ffd3c5d6d839 P 102b2e12e0f546e0a4f6402e8bd91481: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 22:59:56.916603 18290 raft_consensus.cc:482] T 8b1b17c6705244ecbf38ffd3c5d6d839 P 1601d737452f4cb5bd908fbd84c183a9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 22:59:56.916953 18290 raft_consensus.cc:504] T 8b1b17c6705244ecbf38ffd3c5d6d839 P 1601d737452f4cb5bd908fbd84c183a9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd086879c54a40bdb4dfecda24a0736b" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 46643 } } peers { permanent_uuid: "102b2e12e0f546e0a4f6402e8bd91481" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 39593 } } peers { permanent_uuid: "1601d737452f4cb5bd908fbd84c183a9" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 34735 } }
I20260807 22:59:56.918893 18290 leader_election.cc:288] T 8b1b17c6705244ecbf38ffd3c5d6d839 P 1601d737452f4cb5bd908fbd84c183a9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd086879c54a40bdb4dfecda24a0736b (127.25.94.1:46643), 102b2e12e0f546e0a4f6402e8bd91481 (127.25.94.2:39593)
I20260807 22:59:56.926281 18279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:56.919573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.3:40515 (local address 127.25.94.2:39593)
0807 22:59:56.919823 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0807 22:59:56.919832 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 22:59:56.919861 (+ 29us) server_negotiation.cc:407] Connection header received
0807 22:59:56.919942 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:59:56.919948 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:59:56.920026 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:59:56.920125 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:59:56.920630 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:59:56.921669 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:59:56.924116 (+ 2447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:59:56.925402 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:59:56.925545 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:59:56.925648 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0807 22:59:56.925931 (+ 283us) server_negotiation.cc:299] Negotiation successful
0807 22:59:56.926027 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260807 22:59:56.926342 18261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:56.919268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.2:39593 (local address 127.25.94.3:40515)
0807 22:59:56.919525 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0807 22:59:56.919548 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0807 22:59:56.919683 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:59:56.920186 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:59:56.920194 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:59:56.920228 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:59:56.920466 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:59:56.920475 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:59:56.921789 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:59:56.921793 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:59:56.923962 (+ 2169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:59:56.923977 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:59:56.925569 (+ 1592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:59:56.925573 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:59:56.925707 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:59:56.925763 (+ 56us) client_negotiation.cc:769] Sending connection context
0807 22:59:56.925871 (+ 108us) client_negotiation.cc:240] Negotiation successful
0807 22:59:56.926012 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89,"mutex_wait_us":88,"spinlock_wait_cycles":26368}
I20260807 22:59:56.926622 18273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:56.919273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.3:33095 (local address 127.25.94.1:46643)
0807 22:59:56.919667 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0807 22:59:56.919673 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 22:59:56.920255 (+ 582us) server_negotiation.cc:407] Connection header received
0807 22:59:56.920398 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:59:56.920401 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:59:56.920456 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:59:56.920526 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:59:56.921057 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:59:56.922018 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:59:56.924362 (+ 2344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:59:56.925645 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:59:56.925774 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:59:56.925878 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0807 22:59:56.926217 (+ 339us) server_negotiation.cc:299] Negotiation successful
0807 22:59:56.926363 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243}
I20260807 22:59:56.926883 18293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:56.919310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.1:46643 (local address 127.25.94.3:33095)
0807 22:59:56.920138 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0807 22:59:56.920155 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 22:59:56.920265 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:59:56.920579 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:59:56.920583 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:59:56.920619 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:59:56.920904 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:59:56.920912 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:59:56.922172 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:59:56.922178 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:59:56.924211 (+ 2033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:59:56.924221 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:59:56.925882 (+ 1661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:59:56.925896 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:59:56.926054 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:59:56.926083 (+ 29us) client_negotiation.cc:769] Sending connection context
0807 22:59:56.926172 (+ 89us) client_negotiation.cc:240] Negotiation successful
0807 22:59:56.926288 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":706,"mutex_wait_us":23,"thread_start_us":528,"threads_started":1}
I20260807 22:59:56.927546 17970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b1b17c6705244ecbf38ffd3c5d6d839" candidate_uuid: "1601d737452f4cb5bd908fbd84c183a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "102b2e12e0f546e0a4f6402e8bd91481" is_pre_election: true
I20260807 22:59:56.927752 17753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b1b17c6705244ecbf38ffd3c5d6d839" candidate_uuid: "1601d737452f4cb5bd908fbd84c183a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd086879c54a40bdb4dfecda24a0736b" is_pre_election: true
I20260807 22:59:56.928071 17970 raft_consensus.cc:2454] T 8b1b17c6705244ecbf38ffd3c5d6d839 P 102b2e12e0f546e0a4f6402e8bd91481 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1601d737452f4cb5bd908fbd84c183a9 in term 0.
I20260807 22:59:56.928332 17753 raft_consensus.cc:2454] T 8b1b17c6705244ecbf38ffd3c5d6d839 P cd086879c54a40bdb4dfecda24a0736b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1601d737452f4cb5bd908fbd84c183a9 in term 0.
I20260807 22:59:56.928793 18071 leader_election.cc:302] T 8b1b17c6705244ecbf38ffd3c5d6d839 P 1601d737452f4cb5bd908fbd84c183a9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 102b2e12e0f546e0a4f6402e8bd91481, 1601d737452f4cb5bd908fbd84c183a9; no voters:
I20260807 22:59:56.929262 18290 raft_consensus.cc:2781] T 8b1b17c6705244ecbf38ffd3c5d6d839 P 1601d737452f4cb5bd908fbd84c183a9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 22:59:56.929432 18290 raft_consensus.cc:482] T 8b1b17c6705244ecbf38ffd3c5d6d839 P 1601d737452f4cb5bd908fbd84c183a9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 22:59:56.929554 18290 raft_consensus.cc:3037] T 8b1b17c6705244ecbf38ffd3c5d6d839 P 1601d737452f4cb5bd908fbd84c183a9 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:59:56.931929 18290 raft_consensus.cc:504] T 8b1b17c6705244ecbf38ffd3c5d6d839 P 1601d737452f4cb5bd908fbd84c183a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd086879c54a40bdb4dfecda24a0736b" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 46643 } } peers { permanent_uuid: "102b2e12e0f546e0a4f6402e8bd91481" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 39593 } } peers { permanent_uuid: "1601d737452f4cb5bd908fbd84c183a9" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 34735 } }
I20260807 22:59:56.932578 18290 leader_election.cc:288] T 8b1b17c6705244ecbf38ffd3c5d6d839 P 1601d737452f4cb5bd908fbd84c183a9 [CANDIDATE]: Term 1 election: Requested vote from peers cd086879c54a40bdb4dfecda24a0736b (127.25.94.1:46643), 102b2e12e0f546e0a4f6402e8bd91481 (127.25.94.2:39593)
I20260807 22:59:56.933058 17970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b1b17c6705244ecbf38ffd3c5d6d839" candidate_uuid: "1601d737452f4cb5bd908fbd84c183a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "102b2e12e0f546e0a4f6402e8bd91481"
I20260807 22:59:56.933058 17753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b1b17c6705244ecbf38ffd3c5d6d839" candidate_uuid: "1601d737452f4cb5bd908fbd84c183a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd086879c54a40bdb4dfecda24a0736b"
I20260807 22:59:56.933305 17970 raft_consensus.cc:3037] T 8b1b17c6705244ecbf38ffd3c5d6d839 P 102b2e12e0f546e0a4f6402e8bd91481 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:59:56.933305 17753 raft_consensus.cc:3037] T 8b1b17c6705244ecbf38ffd3c5d6d839 P cd086879c54a40bdb4dfecda24a0736b [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:59:56.936600 17753 raft_consensus.cc:2454] T 8b1b17c6705244ecbf38ffd3c5d6d839 P cd086879c54a40bdb4dfecda24a0736b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1601d737452f4cb5bd908fbd84c183a9 in term 1.
I20260807 22:59:56.936600 17970 raft_consensus.cc:2454] T 8b1b17c6705244ecbf38ffd3c5d6d839 P 102b2e12e0f546e0a4f6402e8bd91481 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1601d737452f4cb5bd908fbd84c183a9 in term 1.
I20260807 22:59:56.937024 18071 leader_election.cc:302] T 8b1b17c6705244ecbf38ffd3c5d6d839 P 1601d737452f4cb5bd908fbd84c183a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 102b2e12e0f546e0a4f6402e8bd91481, 1601d737452f4cb5bd908fbd84c183a9; no voters:
I20260807 22:59:56.937348 18290 raft_consensus.cc:2781] T 8b1b17c6705244ecbf38ffd3c5d6d839 P 1601d737452f4cb5bd908fbd84c183a9 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:59:56.937873 18290 raft_consensus.cc:686] T 8b1b17c6705244ecbf38ffd3c5d6d839 P 1601d737452f4cb5bd908fbd84c183a9 [term 1 LEADER]: Becoming Leader. State: Replica: 1601d737452f4cb5bd908fbd84c183a9, State: Running, Role: LEADER
I20260807 22:59:56.938362 18290 consensus_queue.cc:237] T 8b1b17c6705244ecbf38ffd3c5d6d839 P 1601d737452f4cb5bd908fbd84c183a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd086879c54a40bdb4dfecda24a0736b" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 46643 } } peers { permanent_uuid: "102b2e12e0f546e0a4f6402e8bd91481" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 39593 } } peers { permanent_uuid: "1601d737452f4cb5bd908fbd84c183a9" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 34735 } }
I20260807 22:59:56.943531 17510 catalog_manager.cc:5495] T 8b1b17c6705244ecbf38ffd3c5d6d839 P 1601d737452f4cb5bd908fbd84c183a9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1601d737452f4cb5bd908fbd84c183a9 (127.25.94.3). New cstate: current_term: 1 leader_uuid: "1601d737452f4cb5bd908fbd84c183a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd086879c54a40bdb4dfecda24a0736b" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 46643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "102b2e12e0f546e0a4f6402e8bd91481" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 39593 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1601d737452f4cb5bd908fbd84c183a9" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 34735 } health_report { overall_health: HEALTHY } } }
I20260807 22:59:57.013619 18280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:57.008030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50300 (local address 127.25.94.3:34735)
0807 22:59:57.008259 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0807 22:59:57.008265 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 22:59:57.008286 (+ 21us) server_negotiation.cc:407] Connection header received
0807 22:59:57.008451 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:59:57.008455 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:59:57.008544 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:59:57.008709 (+ 165us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 22:59:57.009080 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:59:57.009960 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:59:57.011365 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:59:57.011772 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:59:57.011926 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:59:57.012449 (+ 523us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 22:59:57.012964 (+ 515us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 22:59:57.013124 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0807 22:59:57.013273 (+ 149us) server_negotiation.cc:299] Negotiation successful
0807 22:59:57.013354 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260807 22:59:57.022070 17753 raft_consensus.cc:1261] T 8b1b17c6705244ecbf38ffd3c5d6d839 P cd086879c54a40bdb4dfecda24a0736b [term 1 FOLLOWER]: Refusing update from remote peer 1601d737452f4cb5bd908fbd84c183a9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 22:59:57.022068 17970 raft_consensus.cc:1261] T 8b1b17c6705244ecbf38ffd3c5d6d839 P 102b2e12e0f546e0a4f6402e8bd91481 [term 1 FOLLOWER]: Refusing update from remote peer 1601d737452f4cb5bd908fbd84c183a9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 22:59:57.023109 18294 consensus_queue.cc:1035] T 8b1b17c6705244ecbf38ffd3c5d6d839 P 1601d737452f4cb5bd908fbd84c183a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "102b2e12e0f546e0a4f6402e8bd91481" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 39593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 22:59:57.023497 18290 consensus_queue.cc:1035] T 8b1b17c6705244ecbf38ffd3c5d6d839 P 1601d737452f4cb5bd908fbd84c183a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd086879c54a40bdb4dfecda24a0736b" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 46643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 22:59:57.054689 17510 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:51502:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 22:59:57.055016 17510 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:51502:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 22:59:57.059072 17510 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 70cb141317c44ced96448d499c69dd93: Sending DeleteTablet for 3 replicas of tablet 8b1b17c6705244ecbf38ffd3c5d6d839
I20260807 22:59:57.060503 17935 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b1b17c6705244ecbf38ffd3c5d6d839 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:59:57 UTC) from {username='kudu', principal='kudu/127.25.94.62@KRBTEST.COM'} at 127.0.0.1:50312
I20260807 22:59:57.060786 18146 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b1b17c6705244ecbf38ffd3c5d6d839 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:59:57 UTC) from {username='kudu', principal='kudu/127.25.94.62@KRBTEST.COM'} at 127.0.0.1:50292
I20260807 22:59:57.061414 17720 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b1b17c6705244ecbf38ffd3c5d6d839 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:59:57 UTC) from {username='kudu', principal='kudu/127.25.94.62@KRBTEST.COM'} at 127.0.0.1:51270
I20260807 22:59:57.061769 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 17576
I20260807 22:59:57.061990 18314 tablet_replica.cc:330] T 8b1b17c6705244ecbf38ffd3c5d6d839 P 1601d737452f4cb5bd908fbd84c183a9: stopping tablet replica
I20260807 22:59:57.063035 18313 tablet_replica.cc:330] T 8b1b17c6705244ecbf38ffd3c5d6d839 P 102b2e12e0f546e0a4f6402e8bd91481: stopping tablet replica
I20260807 22:59:57.062988 18314 raft_consensus.cc:2229] T 8b1b17c6705244ecbf38ffd3c5d6d839 P 1601d737452f4cb5bd908fbd84c183a9 [term 1 LEADER]: Raft consensus shutting down.
I20260807 22:59:57.063452 18313 raft_consensus.cc:2229] T 8b1b17c6705244ecbf38ffd3c5d6d839 P 102b2e12e0f546e0a4f6402e8bd91481 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 22:59:57.064055 18313 raft_consensus.cc:2258] T 8b1b17c6705244ecbf38ffd3c5d6d839 P 102b2e12e0f546e0a4f6402e8bd91481 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 22:59:57.064122 18314 raft_consensus.cc:2258] T 8b1b17c6705244ecbf38ffd3c5d6d839 P 1601d737452f4cb5bd908fbd84c183a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 22:59:57.066359 18313 ts_tablet_manager.cc:1905] T 8b1b17c6705244ecbf38ffd3c5d6d839 P 102b2e12e0f546e0a4f6402e8bd91481: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 22:59:57.066660 18314 ts_tablet_manager.cc:1905] T 8b1b17c6705244ecbf38ffd3c5d6d839 P 1601d737452f4cb5bd908fbd84c183a9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 22:59:57.073685 18313 ts_tablet_manager.cc:1918] T 8b1b17c6705244ecbf38ffd3c5d6d839 P 102b2e12e0f546e0a4f6402e8bd91481: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 22:59:57.073810 18315 tablet_replica.cc:330] T 8b1b17c6705244ecbf38ffd3c5d6d839 P cd086879c54a40bdb4dfecda24a0736b: stopping tablet replica
I20260807 22:59:57.073941 18313 log.cc:1196] T 8b1b17c6705244ecbf38ffd3c5d6d839 P 102b2e12e0f546e0a4f6402e8bd91481: Deleting WAL directory at /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-1/wal/wals/8b1b17c6705244ecbf38ffd3c5d6d839
I20260807 22:59:57.074561 18313 ts_tablet_manager.cc:1939] T 8b1b17c6705244ecbf38ffd3c5d6d839 P 102b2e12e0f546e0a4f6402e8bd91481: Deleting consensus metadata
I20260807 22:59:57.074595 18315 raft_consensus.cc:2229] T 8b1b17c6705244ecbf38ffd3c5d6d839 P cd086879c54a40bdb4dfecda24a0736b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 22:59:57.075387 18315 raft_consensus.cc:2258] T 8b1b17c6705244ecbf38ffd3c5d6d839 P cd086879c54a40bdb4dfecda24a0736b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 22:59:57.076721 17483 catalog_manager.cc:4841] TS 102b2e12e0f546e0a4f6402e8bd91481 (127.25.94.2:39593): tablet 8b1b17c6705244ecbf38ffd3c5d6d839 (table test-table [id=e1fb3c2a02804b96b5ffaa177144e4c6]) successfully deleted
I20260807 22:59:57.078323 18314 ts_tablet_manager.cc:1918] T 8b1b17c6705244ecbf38ffd3c5d6d839 P 1601d737452f4cb5bd908fbd84c183a9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 22:59:57.078548 18314 log.cc:1196] T 8b1b17c6705244ecbf38ffd3c5d6d839 P 1601d737452f4cb5bd908fbd84c183a9: Deleting WAL directory at /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-2/wal/wals/8b1b17c6705244ecbf38ffd3c5d6d839
I20260807 22:59:57.078679 18315 ts_tablet_manager.cc:1905] T 8b1b17c6705244ecbf38ffd3c5d6d839 P cd086879c54a40bdb4dfecda24a0736b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 22:59:57.079128 18314 ts_tablet_manager.cc:1939] T 8b1b17c6705244ecbf38ffd3c5d6d839 P 1601d737452f4cb5bd908fbd84c183a9: Deleting consensus metadata
I20260807 22:59:57.080967 17478 catalog_manager.cc:4841] TS 1601d737452f4cb5bd908fbd84c183a9 (127.25.94.3:34735): tablet 8b1b17c6705244ecbf38ffd3c5d6d839 (table test-table [id=e1fb3c2a02804b96b5ffaa177144e4c6]) successfully deleted
I20260807 22:59:57.081040 18273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:57.062702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51284 (local address 127.25.94.1:46643)
0807 22:59:57.062965 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0807 22:59:57.062973 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 22:59:57.063042 (+ 69us) server_negotiation.cc:407] Connection header received
0807 22:59:57.063268 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:59:57.063274 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:59:57.063399 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:59:57.063516 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:59:57.065635 (+ 2119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:59:57.066805 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:59:57.068761 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:59:57.069495 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:59:57.069615 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:59:57.073185 (+ 3570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:59:57.073204 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:59:57.073224 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:59:57.073275 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:59:57.077514 (+ 4239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:59:57.078437 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:59:57.078449 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:59:57.078452 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:59:57.078660 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:59:57.079190 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:59:57.079196 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:59:57.079200 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:59:57.079676 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:59:57.080061 (+ 385us) server_negotiation.cc:1035] Waiting for connection context
0807 22:59:57.080364 (+ 303us) server_negotiation.cc:299] Negotiation successful
0807 22:59:57.080616 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260807 22:59:57.081833 17815 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 22:59:57.207055 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 17576
I20260807 22:59:57.240963 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 17831
W20260807 22:59:57.240962 17487 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.25.94.1:46643 (error 108)
I20260807 22:59:57.243251 18318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:57.242372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.1:46643 (local address 127.0.0.1:51292)
0807 22:59:57.242873 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0807 22:59:57.243016 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.1:46643: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":225,"threads_started":1}
W20260807 22:59:57.243539 17487 catalog_manager.cc:4584] TS cd086879c54a40bdb4dfecda24a0736b (127.25.94.1:46643): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8b1b17c6705244ecbf38ffd3c5d6d839: Network error: Client connection negotiation failed: client connection to 127.25.94.1:46643: connect: Connection refused (error 111)
I20260807 22:59:57.259068 18279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:57.242306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50328 (local address 127.25.94.2:39593)
0807 22:59:57.242508 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0807 22:59:57.242517 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 22:59:57.242542 (+ 25us) server_negotiation.cc:407] Connection header received
0807 22:59:57.242634 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:59:57.242642 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:59:57.242749 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:59:57.242906 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:59:57.244496 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:59:57.245591 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:59:57.247734 (+ 2143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:59:57.248226 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:59:57.248351 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:59:57.252241 (+ 3890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:59:57.252259 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:59:57.252277 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:59:57.252319 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:59:57.255503 (+ 3184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:59:57.256502 (+ 999us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:59:57.256510 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:59:57.256512 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:59:57.256643 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:59:57.257132 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:59:57.257137 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:59:57.257139 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:59:57.257733 (+ 594us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:59:57.257965 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0807 22:59:57.258415 (+ 450us) server_negotiation.cc:299] Negotiation successful
0807 22:59:57.258644 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260807 22:59:57.259856 18036 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 22:59:57.297624 18318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:57.296978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.1:46643 (local address 127.0.0.1:51300)
0807 22:59:57.297300 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0807 22:59:57.297427 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.1:46643: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
I20260807 22:59:57.298905 18318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:57.298471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.1:46643 (local address 127.0.0.1:51310)
0807 22:59:57.298642 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0807 22:59:57.298737 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.1:46643: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260807 22:59:57.353875 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 17831
I20260807 22:59:57.359721 18318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:57.359093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.1:46643 (local address 127.0.0.1:51320)
0807 22:59:57.359402 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0807 22:59:57.359518 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.1:46643: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164}
I20260807 22:59:57.362295 18318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:57.360444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.1:46643 (local address 127.0.0.1:51330)
0807 22:59:57.361995 (+ 1551us) negotiation.cc:107] Waiting for socket to connect
0807 22:59:57.362099 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.1:46643: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1448,"threads_started":1}
I20260807 22:59:57.382632 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 18045
I20260807 22:59:57.401527 18280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:57.384166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50306 (local address 127.25.94.3:34735)
0807 22:59:57.384486 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0807 22:59:57.384495 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 22:59:57.384520 (+ 25us) server_negotiation.cc:407] Connection header received
0807 22:59:57.384611 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:59:57.384617 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:59:57.384726 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:59:57.384859 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:59:57.387123 (+ 2264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:59:57.388427 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:59:57.389965 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:59:57.390700 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:59:57.390861 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:59:57.393812 (+ 2951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:59:57.393829 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:59:57.393849 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:59:57.393894 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:59:57.398217 (+ 4323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:59:57.399085 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:59:57.399095 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:59:57.399099 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:59:57.399319 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:59:57.399719 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:59:57.399725 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:59:57.399729 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:59:57.400254 (+ 525us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:59:57.400568 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0807 22:59:57.400791 (+ 223us) server_negotiation.cc:299] Negotiation successful
0807 22:59:57.401019 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260807 22:59:57.402446 18243 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 22:59:57.451021 18320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:57.450340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.1:46643 (local address 127.0.0.1:51334)
0807 22:59:57.450622 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0807 22:59:57.450796 (+ 174us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.1:46643: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260807 22:59:57.452250 18320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:57.451793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.1:46643 (local address 127.0.0.1:51348)
0807 22:59:57.452013 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0807 22:59:57.452108 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.1:46643: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260807 22:59:57.527254 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 18045
I20260807 22:59:57.558332 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 17451
I20260807 22:59:57.559553 17567 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 22:59:57.673311 18320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:57.672612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.1:46643 (local address 127.0.0.1:51354)
0807 22:59:57.672879 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0807 22:59:57.672997 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.1:46643: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260807 22:59:57.673332 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 17451
I20260807 22:59:57.711795 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.62:42307
--webserver_interface=127.25.94.62
--webserver_port=42987
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.62.keytab
--principal=kudu/127.25.94.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.20:41671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.94.62:42307
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:59:58.074502 18330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:59:58.074796 18330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:59:58.074966 18330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:59:58.082069 18330 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 22:59:58.082188 18330 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:59:58.082300 18330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:59:58.082402 18330 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 22:59:58.082453 18330 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 22:59:58.092932 18330 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:41671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.94.62:42307
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.62.keytab
--principal=kudu/127.25.94.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.62:42307
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.94.62
--webserver_port=42987
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 22:59:58.094384 18330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:59:58.095741 18330 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:59:58.108620 18340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:59:58.108587 18344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:59:58.108637 18342 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:59:58.110083 18330 server_base.cc:1034] running on GCE node
I20260807 22:59:58.111117 18330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:59:58.112679 18330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:59:58.113960 18330 hybrid_clock.cc:648] HybridClock initialized: now 1786143598113914 us; error 74 us; skew 500 ppm
I20260807 22:59:58.114485 18330 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 22:59:58 dist-test-slave-z8x9 krb5kdc[16525](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143598, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:59:58.118110 18330 init.cc:377] Logged in from keytab as kudu/127.25.94.62@KRBTEST.COM (short username kudu)
I20260807 22:59:58.119808 18330 webserver.cc:460] Webserver started at http://127.25.94.62:42987/ using document root <none> and password file <none>
I20260807 22:59:58.120595 18330 fs_manager.cc:359] Metadata directory not provided
I20260807 22:59:58.120759 18330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:59:58.126685 18330 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260807 22:59:58.129855 18354 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:59:58.131323 18330 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 22:59:58.131543 18330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/master-0/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/master-0/wal
uuid: "70cb141317c44ced96448d499c69dd93"
format_stamp: "Formatted at 2026-08-07 22:59:51 on dist-test-slave-z8x9"
server_key: "a88c9b3689a8ca5be384dd3b9d50679a"
server_key_iv: "81694a9a1d3615340b77eca7ccef72cd"
server_key_version: "encryptionkey@0"
I20260807 22:59:58.132616 18330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:59:58.166450 18330 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:59:58.172555 18330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:59:58.172904 18330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:59:58.194060 18330 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.62:42307
I20260807 22:59:58.194201 18438 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.62:42307 every 8 connection(s)
I20260807 22:59:58.195854 18330 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
I20260807 22:59:58.198897 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 18330
I20260807 22:59:58.199900 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.1:46643
--local_ip_for_outbound_sockets=127.25.94.1
--tserver_master_addrs=127.25.94.62:42307
--webserver_port=36475
--webserver_interface=127.25.94.1
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.1.keytab
--principal=kudu/127.25.94.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.20:41671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 22:59:58.204303 18439 sys_catalog.cc:263] Verifying existing consensus state
I20260807 22:59:58.208904 18439 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70cb141317c44ced96448d499c69dd93: Bootstrap starting.
I20260807 22:59:58.240535 18439 log.cc:824] T 00000000000000000000000000000000 P 70cb141317c44ced96448d499c69dd93: Log is configured to *not* fsync() on all Append() calls
I20260807 22:59:58.264039 18439 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70cb141317c44ced96448d499c69dd93: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 22:59:58.264772 18439 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70cb141317c44ced96448d499c69dd93: Bootstrap complete.
I20260807 22:59:58.280443 18439 raft_consensus.cc:348] T 00000000000000000000000000000000 P 70cb141317c44ced96448d499c69dd93 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70cb141317c44ced96448d499c69dd93" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 42307 } }
I20260807 22:59:58.282099 18439 raft_consensus.cc:729] T 00000000000000000000000000000000 P 70cb141317c44ced96448d499c69dd93 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70cb141317c44ced96448d499c69dd93, State: Initialized, Role: FOLLOWER
I20260807 22:59:58.282919 18439 consensus_queue.cc:260] T 00000000000000000000000000000000 P 70cb141317c44ced96448d499c69dd93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70cb141317c44ced96448d499c69dd93" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 42307 } }
I20260807 22:59:58.283217 18439 raft_consensus.cc:388] T 00000000000000000000000000000000 P 70cb141317c44ced96448d499c69dd93 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 22:59:58.283363 18439 raft_consensus.cc:482] T 00000000000000000000000000000000 P 70cb141317c44ced96448d499c69dd93 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 22:59:58.283560 18439 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 70cb141317c44ced96448d499c69dd93 [term 2 FOLLOWER]: Advancing to term 3
I20260807 22:59:58.287956 18439 raft_consensus.cc:504] T 00000000000000000000000000000000 P 70cb141317c44ced96448d499c69dd93 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70cb141317c44ced96448d499c69dd93" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 42307 } }
I20260807 22:59:58.288550 18439 leader_election.cc:302] T 00000000000000000000000000000000 P 70cb141317c44ced96448d499c69dd93 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 70cb141317c44ced96448d499c69dd93; no voters:
I20260807 22:59:58.289290 18439 leader_election.cc:288] T 00000000000000000000000000000000 P 70cb141317c44ced96448d499c69dd93 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260807 22:59:58.289472 18447 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 70cb141317c44ced96448d499c69dd93 [term 3 FOLLOWER]: Leader election won for term 3
I20260807 22:59:58.290681 18447 raft_consensus.cc:686] T 00000000000000000000000000000000 P 70cb141317c44ced96448d499c69dd93 [term 3 LEADER]: Becoming Leader. State: Replica: 70cb141317c44ced96448d499c69dd93, State: Running, Role: LEADER
I20260807 22:59:58.291266 18447 consensus_queue.cc:237] T 00000000000000000000000000000000 P 70cb141317c44ced96448d499c69dd93 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70cb141317c44ced96448d499c69dd93" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 42307 } }
I20260807 22:59:58.291501 18439 sys_catalog.cc:564] T 00000000000000000000000000000000 P 70cb141317c44ced96448d499c69dd93 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 22:59:58.294049 18450 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70cb141317c44ced96448d499c69dd93 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 70cb141317c44ced96448d499c69dd93. Latest consensus state: current_term: 3 leader_uuid: "70cb141317c44ced96448d499c69dd93" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70cb141317c44ced96448d499c69dd93" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 42307 } } }
I20260807 22:59:58.294534 18450 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70cb141317c44ced96448d499c69dd93 [sys.catalog]: This master's current role is: LEADER
I20260807 22:59:58.295192 18449 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70cb141317c44ced96448d499c69dd93 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "70cb141317c44ced96448d499c69dd93" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70cb141317c44ced96448d499c69dd93" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 42307 } } }
I20260807 22:59:58.295532 18449 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70cb141317c44ced96448d499c69dd93 [sys.catalog]: This master's current role is: LEADER
I20260807 22:59:58.305605 18458 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 22:59:58.316527 18458 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 22:59:58.318543 18458 catalog_manager.cc:1259] Loaded cluster ID: 3e96fff9e7a54bfabf8a7aca3984e5c7
I20260807 22:59:58.318742 18458 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260807 22:59:58.321359 18458 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
I20260807 22:59:58.323602 18471 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
@ 0x7fb32a60d9ad google::LogMessage::Fail() at ??:0
@ 0x7fb32a61088a google::LogMessage::SendToLog() at ??:0
@ 0x7fb32a60d4de google::LogMessage::Flush() at ??:0
@ 0x7fb32a60ea19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fb340040512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fb34003d976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fb3400c5610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fb3400c5439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fb32d259345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fb32d25c91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fb32d25c799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fb32d237eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fb3305116db start_thread at ??:0
@ 0x7fb3294a761f clone at ??:0
W20260807 22:59:58.638248 18443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:59:58.638540 18443 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:59:58.638710 18443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:59:58.646049 18443 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:59:58.646169 18443 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:59:58.646359 18443 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.1
I20260807 22:59:58.656216 18443 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:41671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.1.keytab
--principal=kudu/127.25.94.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.1:46643
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.94.1
--webserver_port=36475
--tserver_master_addrs=127.25.94.62:42307
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.1
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 22:59:58.657964 18443 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:59:58.659296 18443 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:59:58.675575 18505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:59:58.675560 18509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:59:58.675689 18504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:59:58.676466 18443 server_base.cc:1034] running on GCE node
I20260807 22:59:58.677331 18443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:59:58.678422 18443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:59:58.679708 18443 hybrid_clock.cc:648] HybridClock initialized: now 1786143598679646 us; error 55 us; skew 500 ppm
I20260807 22:59:58.680083 18443 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 22:59:58 dist-test-slave-z8x9 krb5kdc[16525](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143598, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:59:58.683642 18443 init.cc:377] Logged in from keytab as kudu/127.25.94.1@KRBTEST.COM (short username kudu)
I20260807 22:59:58.685336 18443 webserver.cc:460] Webserver started at http://127.25.94.1:36475/ using document root <none> and password file <none>
I20260807 22:59:58.686334 18443 fs_manager.cc:359] Metadata directory not provided
I20260807 22:59:58.686460 18443 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:59:58.692426 18443 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260807 22:59:58.695660 18522 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:59:58.697141 18443 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 22:59:58.697469 18443 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-0/wal
uuid: "cd086879c54a40bdb4dfecda24a0736b"
format_stamp: "Formatted at 2026-08-07 22:59:52 on dist-test-slave-z8x9"
server_key: "bd0710249b124060f98c5a18ebee52b4"
server_key_iv: "b38ebd9c39be82327b298dc096e71404"
server_key_version: "encryptionkey@0"
I20260807 22:59:58.698516 18443 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:59:58.723251 18443 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:59:58.727792 18443 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:59:58.728084 18443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:59:58.729184 18443 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:59:58.731062 18534 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260807 22:59:58.736692 18443 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260807 22:59:58.736851 18443 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260807 22:59:58.736995 18443 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260807 22:59:58.740094 18443 ts_tablet_manager.cc:610] Registered 1 tablets
I20260807 22:59:58.740208 18443 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.001s sys 0.000s
I20260807 22:59:58.740607 18534 tablet_bootstrap.cc:492] T 8b1b17c6705244ecbf38ffd3c5d6d839 P cd086879c54a40bdb4dfecda24a0736b: Bootstrap starting.
I20260807 22:59:58.749614 18534 log.cc:824] T 8b1b17c6705244ecbf38ffd3c5d6d839 P cd086879c54a40bdb4dfecda24a0736b: Log is configured to *not* fsync() on all Append() calls
I20260807 22:59:58.768849 18534 tablet_bootstrap.cc:492] T 8b1b17c6705244ecbf38ffd3c5d6d839 P cd086879c54a40bdb4dfecda24a0736b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 22:59:58.769871 18534 tablet_bootstrap.cc:492] T 8b1b17c6705244ecbf38ffd3c5d6d839 P cd086879c54a40bdb4dfecda24a0736b: Bootstrap complete.
I20260807 22:59:58.770542 18534 ts_tablet_manager.cc:1397] T 8b1b17c6705244ecbf38ffd3c5d6d839 P cd086879c54a40bdb4dfecda24a0736b: Time spent bootstrapping tablet: real 0.030s user 0.028s sys 0.000s
I20260807 22:59:58.779670 18443 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.1:46643
I20260807 22:59:58.779695 18709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.1:46643 every 8 connection(s)
I20260807 22:59:58.781524 18443 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-0/data/info.pb
I20260807 22:59:58.783617 18534 raft_consensus.cc:348] T 8b1b17c6705244ecbf38ffd3c5d6d839 P cd086879c54a40bdb4dfecda24a0736b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd086879c54a40bdb4dfecda24a0736b" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 46643 } } peers { permanent_uuid: "102b2e12e0f546e0a4f6402e8bd91481" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 39593 } } peers { permanent_uuid: "1601d737452f4cb5bd908fbd84c183a9" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 34735 } }
I20260807 22:59:58.784956 18534 raft_consensus.cc:729] T 8b1b17c6705244ecbf38ffd3c5d6d839 P cd086879c54a40bdb4dfecda24a0736b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd086879c54a40bdb4dfecda24a0736b, State: Initialized, Role: FOLLOWER
I20260807 22:59:58.785818 18534 consensus_queue.cc:260] T 8b1b17c6705244ecbf38ffd3c5d6d839 P cd086879c54a40bdb4dfecda24a0736b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd086879c54a40bdb4dfecda24a0736b" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 46643 } } peers { permanent_uuid: "102b2e12e0f546e0a4f6402e8bd91481" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 39593 } } peers { permanent_uuid: "1601d737452f4cb5bd908fbd84c183a9" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 34735 } }
I20260807 22:59:58.787352 18534 ts_tablet_manager.cc:1428] T 8b1b17c6705244ecbf38ffd3c5d6d839 P cd086879c54a40bdb4dfecda24a0736b: Time spent starting tablet: real 0.017s user 0.017s sys 0.000s
I20260807 22:59:58.787446 18712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:58.784795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:42307 (local address 127.25.94.1:34773)
0807 22:59:58.785945 (+ 1150us) negotiation.cc:107] Waiting for socket to connect
0807 22:59:58.786267 (+ 322us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.62:42307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":842,"thread_start_us":529,"threads_started":1}
W20260807 22:59:58.788151 18710 heartbeater.cc:644] Failed to heartbeat to 127.25.94.62:42307 (0 consecutive failures): Network error: Failed to ping master at 127.25.94.62:42307: Client connection negotiation failed: client connection to 127.25.94.62:42307: connect: Connection refused (error 111)
I20260807 22:59:58.789064 18712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:58.788702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:42307 (local address 127.25.94.1:60833)
0807 22:59:58.788879 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0807 22:59:58.788956 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.62:42307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260807 22:59:58.789467 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 18443
I20260807 22:59:58.789902 18712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:58.789556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:42307 (local address 127.25.94.1:48159)
0807 22:59:58.789701 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0807 22:59:58.789786 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.62:42307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260807 22:59:58.790122 18710 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260807 22:59:58.790205 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.2:39593
--local_ip_for_outbound_sockets=127.25.94.2
--tserver_master_addrs=127.25.94.62:42307
--webserver_port=42255
--webserver_interface=127.25.94.2
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.2.keytab
--principal=kudu/127.25.94.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.20:41671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:59:59.130777 18715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:59:59.131089 18715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:59:59.131263 18715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:59:59.138724 18715 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:59:59.138849 18715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:59:59.139035 18715 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.2
I20260807 22:59:59.149039 18715 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:41671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.2.keytab
--principal=kudu/127.25.94.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.2:39593
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.94.2
--webserver_port=42255
--tserver_master_addrs=127.25.94.62:42307
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.2
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 22:59:59.150547 18715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:59:59.151968 18715 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:59:59.166428 18727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:59:59.166324 18729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:59:59.166436 18732 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:59:59.168085 18715 server_base.cc:1034] running on GCE node
I20260807 22:59:59.168821 18715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:59:59.170001 18715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:59:59.171265 18715 hybrid_clock.cc:648] HybridClock initialized: now 1786143599171182 us; error 66 us; skew 500 ppm
I20260807 22:59:59.171653 18715 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 22:59:59 dist-test-slave-z8x9 krb5kdc[16525](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143599, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:59:59.175338 18715 init.cc:377] Logged in from keytab as kudu/127.25.94.2@KRBTEST.COM (short username kudu)
I20260807 22:59:59.177256 18715 webserver.cc:460] Webserver started at http://127.25.94.2:42255/ using document root <none> and password file <none>
I20260807 22:59:59.178047 18715 fs_manager.cc:359] Metadata directory not provided
I20260807 22:59:59.178215 18715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:59:59.184356 18715 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.002s
I20260807 22:59:59.188019 18743 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:59:59.189657 18715 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260807 22:59:59.189864 18715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-1/wal
uuid: "102b2e12e0f546e0a4f6402e8bd91481"
format_stamp: "Formatted at 2026-08-07 22:59:52 on dist-test-slave-z8x9"
server_key: "e857eaed6f47aa89a95c92b0d991ecae"
server_key_iv: "73e5cbde04ab8fe59499c8755f05edcd"
server_key_version: "encryptionkey@0"
I20260807 22:59:59.190954 18715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:59:59.207850 18715 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:59:59.212683 18715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:59:59.213027 18715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:59:59.214212 18715 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:59:59.215754 18715 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:59:59.215893 18715 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:59:59.216037 18715 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:59:59.216135 18715 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:59:59.252671 18715 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.2:39593
I20260807 22:59:59.252768 18931 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.2:39593 every 8 connection(s)
I20260807 22:59:59.254338 18715 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-1/data/info.pb
I20260807 22:59:59.259398 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 18715
I20260807 22:59:59.259449 18936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:59.257412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:42307 (local address 127.25.94.2:48727)
0807 22:59:59.258333 (+ 921us) negotiation.cc:107] Waiting for socket to connect
0807 22:59:59.258535 (+ 202us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.62:42307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":591,"thread_start_us":396,"threads_started":1}
W20260807 22:59:59.260326 18934 heartbeater.cc:644] Failed to heartbeat to 127.25.94.62:42307 (0 consecutive failures): Network error: Failed to ping master at 127.25.94.62:42307: Client connection negotiation failed: client connection to 127.25.94.62:42307: connect: Connection refused (error 111)
I20260807 22:59:59.260346 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.3:34735
--local_ip_for_outbound_sockets=127.25.94.3
--tserver_master_addrs=127.25.94.62:42307
--webserver_port=39035
--webserver_interface=127.25.94.3
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.3.keytab
--principal=kudu/127.25.94.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.20:41671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 22:59:59.261353 18936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:59.260932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:42307 (local address 127.25.94.2:57599)
0807 22:59:59.261094 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0807 22:59:59.261223 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.62:42307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260807 22:59:59.262212 18936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:59.261888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:42307 (local address 127.25.94.2:48017)
0807 22:59:59.262030 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0807 22:59:59.262109 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.62:42307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260807 22:59:59.262481 18934 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260807 22:59:59.605126 18937 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:59:59.605454 18937 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:59:59.605603 18937 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:59:59.612986 18937 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:59:59.613111 18937 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:59:59.613319 18937 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.3
I20260807 22:59:59.623819 18937 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:41671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.3.keytab
--principal=kudu/127.25.94.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.3:34735
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.94.3
--webserver_port=39035
--tserver_master_addrs=127.25.94.62:42307
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.3
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 22:59:59.625182 18937 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:59:59.626523 18937 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:59:59.639398 18943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:59:59.639478 18945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:59:59.639442 18947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:59:59.640640 18937 server_base.cc:1034] running on GCE node
I20260807 22:59:59.641278 18937 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:59:59.642161 18937 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:59:59.643368 18937 hybrid_clock.cc:648] HybridClock initialized: now 1786143599643314 us; error 52 us; skew 500 ppm
I20260807 22:59:59.643638 18937 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 22:59:59 dist-test-slave-z8x9 krb5kdc[16525](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143599, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:59:59.647032 18937 init.cc:377] Logged in from keytab as kudu/127.25.94.3@KRBTEST.COM (short username kudu)
I20260807 22:59:59.648649 18937 webserver.cc:460] Webserver started at http://127.25.94.3:39035/ using document root <none> and password file <none>
I20260807 22:59:59.649538 18937 fs_manager.cc:359] Metadata directory not provided
I20260807 22:59:59.649654 18937 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:59:59.655264 18937 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260807 22:59:59.658358 18962 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:59:59.660090 18937 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 22:59:59.660264 18937 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-2/wal
uuid: "1601d737452f4cb5bd908fbd84c183a9"
format_stamp: "Formatted at 2026-08-07 22:59:53 on dist-test-slave-z8x9"
server_key: "5269a922dcf25da48da9261b5886e089"
server_key_iv: "ff26757ea98e869d4258b576bf912e45"
server_key_version: "encryptionkey@0"
I20260807 22:59:59.661326 18937 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:59:59.681726 18937 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:59:59.686074 18937 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:59:59.686353 18937 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:59:59.687417 18937 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:59:59.688759 18937 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:59:59.688882 18937 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:59:59.689033 18937 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:59:59.689131 18937 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:59:59.722543 18937 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.3:34735
I20260807 22:59:59.722688 19140 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.3:34735 every 8 connection(s)
I20260807 22:59:59.724066 18937 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786143361779964-25976-0/minicluster-data/ts-2/data/info.pb
I20260807 22:59:59.728780 19144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:59.727022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:42307 (local address 127.25.94.3:43591)
0807 22:59:59.727720 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0807 22:59:59.727911 (+ 191us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.62:42307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":271,"threads_started":1}
W20260807 22:59:59.729539 19142 heartbeater.cc:644] Failed to heartbeat to 127.25.94.62:42307 (0 consecutive failures): Network error: Failed to ping master at 127.25.94.62:42307: Client connection negotiation failed: client connection to 127.25.94.62:42307: connect: Connection refused (error 111)
I20260807 22:59:59.730345 19144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:59.730024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:42307 (local address 127.25.94.3:35949)
0807 22:59:59.730155 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0807 22:59:59.730219 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.62:42307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260807 22:59:59.731086 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 18937
I20260807 22:59:59.731359 19144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:59.730916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:42307 (local address 127.25.94.3:53405)
0807 22:59:59.731077 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0807 22:59:59.731179 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.62:42307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260807 22:59:59.731675 19142 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260807 22:59:59.791539 19146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:59:59.790739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:42307 (local address 127.25.94.1:54605)
0807 22:59:59.791208 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0807 22:59:59.791326 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.62:42307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":226,"threads_started":1}
I20260807 23:00:00.264214 19147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:00:00.263261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:42307 (local address 127.25.94.2:43201)
0807 23:00:00.263792 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0807 23:00:00.263930 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.62:42307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":262,"threads_started":1}
I20260807 23:00:00.642107 19148 raft_consensus.cc:482] T 8b1b17c6705244ecbf38ffd3c5d6d839 P cd086879c54a40bdb4dfecda24a0736b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 23:00:00.642561 19148 raft_consensus.cc:504] T 8b1b17c6705244ecbf38ffd3c5d6d839 P cd086879c54a40bdb4dfecda24a0736b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd086879c54a40bdb4dfecda24a0736b" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 46643 } } peers { permanent_uuid: "102b2e12e0f546e0a4f6402e8bd91481" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 39593 } } peers { permanent_uuid: "1601d737452f4cb5bd908fbd84c183a9" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 34735 } }
I20260807 23:00:00.644356 19148 leader_election.cc:288] T 8b1b17c6705244ecbf38ffd3c5d6d839 P cd086879c54a40bdb4dfecda24a0736b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 102b2e12e0f546e0a4f6402e8bd91481 (127.25.94.2:39593), 1601d737452f4cb5bd908fbd84c183a9 (127.25.94.3:34735)
Aug 07 23:00:00 dist-test-slave-z8x9 krb5kdc[16525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143598, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.1@KRBTEST.COM for kudu/127.25.94.2@KRBTEST.COM
Aug 07 23:00:00 dist-test-slave-z8x9 krb5kdc[16525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143598, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.1@KRBTEST.COM for kudu/127.25.94.3@KRBTEST.COM
I20260807 23:00:00.683691 19154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:00:00.645125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.1:55405 (local address 127.25.94.3:34735)
0807 23:00:00.657832 (+ 12707us) server_negotiation.cc:206] Beginning negotiation
0807 23:00:00.657846 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0807 23:00:00.657882 (+ 36us) server_negotiation.cc:407] Connection header received
0807 23:00:00.660295 (+ 2413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:00:00.660309 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:00:00.660879 (+ 570us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:00:00.661397 (+ 518us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:00:00.664954 (+ 3557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:00:00.667038 (+ 2084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:00:00.670083 (+ 3045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:00:00.670803 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:00:00.671028 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:00:00.676175 (+ 5147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:00:00.676184 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:00:00.676198 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 23:00:00.676246 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:00:00.680010 (+ 3764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:00:00.680900 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:00:00.680906 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:00:00.680911 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:00:00.681087 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:00:00.681545 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:00:00.681552 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:00:00.681559 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:00:00.682034 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:00:00.682228 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0807 23:00:00.682834 (+ 606us) server_negotiation.cc:299] Negotiation successful
0807 23:00:00.683229 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12530,"spinlock_wait_cycles":19712,"thread_start_us":12487,"threads_started":1}
I20260807 23:00:00.683692 19156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:00:00.645100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.1:48113 (local address 127.25.94.2:39593)
0807 23:00:00.657537 (+ 12437us) server_negotiation.cc:206] Beginning negotiation
0807 23:00:00.657556 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0807 23:00:00.657621 (+ 65us) server_negotiation.cc:407] Connection header received
0807 23:00:00.660394 (+ 2773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:00:00.660404 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:00:00.660879 (+ 475us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:00:00.661370 (+ 491us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:00:00.664907 (+ 3537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:00:00.667007 (+ 2100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:00:00.670025 (+ 3018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:00:00.670700 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:00:00.670915 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:00:00.675648 (+ 4733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:00:00.675658 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:00:00.675675 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 23:00:00.675719 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:00:00.679604 (+ 3885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:00:00.680494 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:00:00.680502 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:00:00.680508 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:00:00.680694 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:00:00.681207 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:00:00.681215 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:00:00.681218 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:00:00.681800 (+ 582us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:00:00.682009 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0807 23:00:00.682858 (+ 849us) server_negotiation.cc:299] Negotiation successful
0807 23:00:00.683291 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12235,"thread_start_us":12057,"threads_started":1}
I20260807 23:00:00.683743 19150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:00:00.644692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.2:39593 (local address 127.25.94.1:48113)
0807 23:00:00.657197 (+ 12505us) negotiation.cc:107] Waiting for socket to connect
0807 23:00:00.657218 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 23:00:00.657740 (+ 522us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 23:00:00.661974 (+ 4234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 23:00:00.661991 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 23:00:00.663340 (+ 1349us) client_negotiation.cc:189] Negotiated authn=SASL
0807 23:00:00.664626 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:00:00.664659 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:00:00.667283 (+ 2624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:00:00.667289 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:00:00.669833 (+ 2544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:00:00.669853 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:00:00.670921 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:00:00.670929 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:00:00.671202 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:00:00.672014 (+ 812us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 23:00:00.672039 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0807 23:00:00.675225 (+ 3186us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 23:00:00.679787 (+ 4562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 23:00:00.679797 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 23:00:00.679814 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0807 23:00:00.680314 (+ 500us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 23:00:00.680825 (+ 511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 23:00:00.680830 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 23:00:00.680834 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 23:00:00.681035 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 23:00:00.681960 (+ 925us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 23:00:00.681969 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 23:00:00.682298 (+ 329us) client_negotiation.cc:769] Sending connection context
0807 23:00:00.682574 (+ 276us) client_negotiation.cc:240] Negotiation successful
0807 23:00:00.683182 (+ 608us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12361,"mutex_wait_us":195,"spinlock_wait_cycles":52992,"thread_start_us":12535,"threads_started":1}
I20260807 23:00:00.683820 19152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:00:00.644596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.3:34735 (local address 127.25.94.1:55405)
0807 23:00:00.656750 (+ 12154us) negotiation.cc:107] Waiting for socket to connect
0807 23:00:00.656831 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0807 23:00:00.657825 (+ 994us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 23:00:00.662437 (+ 4612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 23:00:00.662446 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 23:00:00.663323 (+ 877us) client_negotiation.cc:189] Negotiated authn=SASL
0807 23:00:00.664626 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:00:00.664659 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:00:00.667338 (+ 2679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:00:00.667350 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:00:00.669848 (+ 2498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:00:00.669867 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:00:00.671049 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:00:00.671056 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:00:00.671305 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:00:00.672218 (+ 913us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 23:00:00.672236 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 23:00:00.675842 (+ 3606us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 23:00:00.680282 (+ 4440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 23:00:00.680288 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 23:00:00.680293 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 23:00:00.680731 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 23:00:00.681241 (+ 510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 23:00:00.681246 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 23:00:00.681250 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 23:00:00.681416 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 23:00:00.682185 (+ 769us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 23:00:00.682190 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 23:00:00.682353 (+ 163us) client_negotiation.cc:769] Sending connection context
0807 23:00:00.682571 (+ 218us) client_negotiation.cc:240] Negotiation successful
0807 23:00:00.683386 (+ 815us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11938,"mutex_wait_us":525,"spinlock_wait_cycles":12288,"thread_start_us":13059,"threads_started":1}
W20260807 23:00:00.687266 18525 leader_election.cc:341] T 8b1b17c6705244ecbf38ffd3c5d6d839 P cd086879c54a40bdb4dfecda24a0736b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 102b2e12e0f546e0a4f6402e8bd91481 (127.25.94.2:39593): Not found: Tablet not found: 8b1b17c6705244ecbf38ffd3c5d6d839
W20260807 23:00:00.688230 18523 leader_election.cc:341] T 8b1b17c6705244ecbf38ffd3c5d6d839 P cd086879c54a40bdb4dfecda24a0736b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1601d737452f4cb5bd908fbd84c183a9 (127.25.94.3:34735): Not found: Tablet not found: 8b1b17c6705244ecbf38ffd3c5d6d839
I20260807 23:00:00.688359 18523 leader_election.cc:302] T 8b1b17c6705244ecbf38ffd3c5d6d839 P cd086879c54a40bdb4dfecda24a0736b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd086879c54a40bdb4dfecda24a0736b; no voters: 102b2e12e0f546e0a4f6402e8bd91481, 1601d737452f4cb5bd908fbd84c183a9
I20260807 23:00:00.688855 19148 raft_consensus.cc:2726] T 8b1b17c6705244ecbf38ffd3c5d6d839 P cd086879c54a40bdb4dfecda24a0736b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 23:00:00.733376 19157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:00:00.732313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:42307 (local address 127.25.94.3:47125)
0807 23:00:00.732865 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0807 23:00:00.733012 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.62:42307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":275,"threads_started":1}
I20260807 23:00:00.792881 19150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:00:00.792463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:42307 (local address 127.25.94.1:59539)
0807 23:00:00.792639 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0807 23:00:00.792737 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.62:42307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260807 23:00:01.266281 19158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:00:01.265503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:42307 (local address 127.25.94.2:44353)
0807 23:00:01.265995 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0807 23:00:01.266084 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.62:42307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":264,"threads_started":1}
I20260807 23:00:01.735219 19160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:00:01.734364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:42307 (local address 127.25.94.3:46321)
0807 23:00:01.734966 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0807 23:00:01.735056 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.62:42307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":347,"threads_started":1}
I20260807 23:00:01.794727 19161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:00:01.793898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:42307 (local address 127.25.94.1:58235)
0807 23:00:01.794429 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0807 23:00:01.794536 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.62:42307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":303,"threads_started":1}
I20260807 23:00:02.268066 19162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:00:02.267339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:42307 (local address 127.25.94.2:40145)
0807 23:00:02.267796 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0807 23:00:02.267886 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.62:42307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":245,"threads_started":1}
I20260807 23:00:02.656410 19165 raft_consensus.cc:482] T 8b1b17c6705244ecbf38ffd3c5d6d839 P cd086879c54a40bdb4dfecda24a0736b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 23:00:02.656601 19165 raft_consensus.cc:504] T 8b1b17c6705244ecbf38ffd3c5d6d839 P cd086879c54a40bdb4dfecda24a0736b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd086879c54a40bdb4dfecda24a0736b" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 46643 } } peers { permanent_uuid: "102b2e12e0f546e0a4f6402e8bd91481" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 39593 } } peers { permanent_uuid: "1601d737452f4cb5bd908fbd84c183a9" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 34735 } }
I20260807 23:00:02.657475 19165 leader_election.cc:288] T 8b1b17c6705244ecbf38ffd3c5d6d839 P cd086879c54a40bdb4dfecda24a0736b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 102b2e12e0f546e0a4f6402e8bd91481 (127.25.94.2:39593), 1601d737452f4cb5bd908fbd84c183a9 (127.25.94.3:34735)
W20260807 23:00:02.658561 18525 leader_election.cc:341] T 8b1b17c6705244ecbf38ffd3c5d6d839 P cd086879c54a40bdb4dfecda24a0736b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 102b2e12e0f546e0a4f6402e8bd91481 (127.25.94.2:39593): Not found: Tablet not found: 8b1b17c6705244ecbf38ffd3c5d6d839
W20260807 23:00:02.658743 18523 leader_election.cc:341] T 8b1b17c6705244ecbf38ffd3c5d6d839 P cd086879c54a40bdb4dfecda24a0736b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1601d737452f4cb5bd908fbd84c183a9 (127.25.94.3:34735): Not found: Tablet not found: 8b1b17c6705244ecbf38ffd3c5d6d839
I20260807 23:00:02.658979 18523 leader_election.cc:302] T 8b1b17c6705244ecbf38ffd3c5d6d839 P cd086879c54a40bdb4dfecda24a0736b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd086879c54a40bdb4dfecda24a0736b; no voters: 102b2e12e0f546e0a4f6402e8bd91481, 1601d737452f4cb5bd908fbd84c183a9
I20260807 23:00:02.659404 19165 raft_consensus.cc:2726] T 8b1b17c6705244ecbf38ffd3c5d6d839 P cd086879c54a40bdb4dfecda24a0736b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 23:00:02.737565 19166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:00:02.736526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:42307 (local address 127.25.94.3:34499)
0807 23:00:02.737212 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0807 23:00:02.737339 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.62:42307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":309,"threads_started":1}
I20260807 23:00:02.796530 19171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:00:02.795648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:42307 (local address 127.25.94.1:59927)
0807 23:00:02.796239 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0807 23:00:02.796334 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.62:42307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":307,"threads_started":1}
I20260807 23:00:03.269919 19172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:00:03.269060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:42307 (local address 127.25.94.2:45597)
0807 23:00:03.269631 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0807 23:00:03.269725 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.62:42307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":304,"threads_started":1}
I20260807 23:00:03.739214 19173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:00:03.738479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:42307 (local address 127.25.94.3:33235)
0807 23:00:03.738918 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0807 23:00:03.739000 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.62:42307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":223,"threads_started":1}
I20260807 23:00:03.798776 19174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:00:03.797942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:42307 (local address 127.25.94.1:40111)
0807 23:00:03.798455 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0807 23:00:03.798569 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.62:42307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":250,"threads_started":1}
I20260807 23:00:04.271679 19175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:00:04.270982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:42307 (local address 127.25.94.2:52959)
0807 23:00:04.271436 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0807 23:00:04.271517 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.62:42307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":229,"threads_started":1}
I20260807 23:00:04.740983 19177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:00:04.740198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:42307 (local address 127.25.94.3:60567)
0807 23:00:04.740701 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0807 23:00:04.740814 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.62:42307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":273,"threads_started":1}
I20260807 23:00:04.800539 19178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:00:04.799805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:42307 (local address 127.25.94.1:50053)
0807 23:00:04.800287 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0807 23:00:04.800375 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.62:42307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":242,"threads_started":1}
I20260807 23:00:05.273658 19179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:00:05.272728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:42307 (local address 127.25.94.2:52551)
0807 23:00:05.273351 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0807 23:00:05.273448 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.62:42307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":305,"threads_started":1}
I20260807 23:00:05.658733 19181 raft_consensus.cc:482] T 8b1b17c6705244ecbf38ffd3c5d6d839 P cd086879c54a40bdb4dfecda24a0736b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 23:00:05.658936 19181 raft_consensus.cc:504] T 8b1b17c6705244ecbf38ffd3c5d6d839 P cd086879c54a40bdb4dfecda24a0736b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd086879c54a40bdb4dfecda24a0736b" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 46643 } } peers { permanent_uuid: "102b2e12e0f546e0a4f6402e8bd91481" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 39593 } } peers { permanent_uuid: "1601d737452f4cb5bd908fbd84c183a9" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 34735 } }
I20260807 23:00:05.659595 19181 leader_election.cc:288] T 8b1b17c6705244ecbf38ffd3c5d6d839 P cd086879c54a40bdb4dfecda24a0736b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 102b2e12e0f546e0a4f6402e8bd91481 (127.25.94.2:39593), 1601d737452f4cb5bd908fbd84c183a9 (127.25.94.3:34735)
W20260807 23:00:05.660692 18523 leader_election.cc:341] T 8b1b17c6705244ecbf38ffd3c5d6d839 P cd086879c54a40bdb4dfecda24a0736b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1601d737452f4cb5bd908fbd84c183a9 (127.25.94.3:34735): Not found: Tablet not found: 8b1b17c6705244ecbf38ffd3c5d6d839
W20260807 23:00:05.660890 18525 leader_election.cc:341] T 8b1b17c6705244ecbf38ffd3c5d6d839 P cd086879c54a40bdb4dfecda24a0736b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 102b2e12e0f546e0a4f6402e8bd91481 (127.25.94.2:39593): Not found: Tablet not found: 8b1b17c6705244ecbf38ffd3c5d6d839
I20260807 23:00:05.660995 18525 leader_election.cc:302] T 8b1b17c6705244ecbf38ffd3c5d6d839 P cd086879c54a40bdb4dfecda24a0736b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd086879c54a40bdb4dfecda24a0736b; no voters: 102b2e12e0f546e0a4f6402e8bd91481, 1601d737452f4cb5bd908fbd84c183a9
I20260807 23:00:05.661258 19181 raft_consensus.cc:2726] T 8b1b17c6705244ecbf38ffd3c5d6d839 P cd086879c54a40bdb4dfecda24a0736b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 23:00:05.742914 19182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:00:05.742091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:42307 (local address 127.25.94.3:59553)
0807 23:00:05.742613 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0807 23:00:05.742722 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.62:42307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":299,"threads_started":1}
I20260807 23:00:05.802793 19183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:00:05.801724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:42307 (local address 127.25.94.1:43165)
0807 23:00:05.802414 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0807 23:00:05.802515 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.62:42307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":592,"thread_start_us":355,"threads_started":1}
I20260807 23:00:06.275725 19185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:00:06.274677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:42307 (local address 127.25.94.2:50315)
0807 23:00:06.275367 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0807 23:00:06.275521 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.62:42307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":283,"threads_started":1}
I20260807 23:00:06.744658 19186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:00:06.743828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:42307 (local address 127.25.94.3:54653)
0807 23:00:06.744389 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0807 23:00:06.744475 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.62:42307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":250,"threads_started":1}
I20260807 23:00:06.804544 19188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:00:06.803779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:42307 (local address 127.25.94.1:54961)
0807 23:00:06.804258 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0807 23:00:06.804342 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.62:42307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":222,"threads_started":1}
I20260807 23:00:07.277784 19189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:00:07.276806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:42307 (local address 127.25.94.2:44985)
0807 23:00:07.277469 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0807 23:00:07.277571 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.62:42307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":400,"threads_started":1}
I20260807 23:00:07.746598 19191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:00:07.745851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:42307 (local address 127.25.94.3:42835)
0807 23:00:07.746282 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0807 23:00:07.746389 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.62:42307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":220,"threads_started":1}
I20260807 23:00:07.806671 19193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:00:07.805654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:42307 (local address 127.25.94.1:41203)
0807 23:00:07.806256 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0807 23:00:07.806379 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.62:42307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":279,"threads_started":1}
I20260807 23:00:08.279968 19194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:00:08.278942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:42307 (local address 127.25.94.2:60413)
0807 23:00:08.279585 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0807 23:00:08.279717 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.62:42307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":334,"threads_started":1}
I20260807 23:00:08.748335 19196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:00:08.747600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:42307 (local address 127.25.94.3:41283)
0807 23:00:08.748080 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0807 23:00:08.748171 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.62:42307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":247,"threads_started":1}
I20260807 23:00:08.808861 19197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:00:08.807914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:42307 (local address 127.25.94.1:56083)
0807 23:00:08.808513 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0807 23:00:08.808649 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.62:42307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":354,"threads_started":1}
I20260807 23:00:09.090672 19200 raft_consensus.cc:482] T 8b1b17c6705244ecbf38ffd3c5d6d839 P cd086879c54a40bdb4dfecda24a0736b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 23:00:09.090965 19200 raft_consensus.cc:504] T 8b1b17c6705244ecbf38ffd3c5d6d839 P cd086879c54a40bdb4dfecda24a0736b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd086879c54a40bdb4dfecda24a0736b" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 46643 } } peers { permanent_uuid: "102b2e12e0f546e0a4f6402e8bd91481" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 39593 } } peers { permanent_uuid: "1601d737452f4cb5bd908fbd84c183a9" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 34735 } }
I20260807 23:00:09.091761 19200 leader_election.cc:288] T 8b1b17c6705244ecbf38ffd3c5d6d839 P cd086879c54a40bdb4dfecda24a0736b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 102b2e12e0f546e0a4f6402e8bd91481 (127.25.94.2:39593), 1601d737452f4cb5bd908fbd84c183a9 (127.25.94.3:34735)
W20260807 23:00:09.092808 18523 leader_election.cc:341] T 8b1b17c6705244ecbf38ffd3c5d6d839 P cd086879c54a40bdb4dfecda24a0736b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1601d737452f4cb5bd908fbd84c183a9 (127.25.94.3:34735): Not found: Tablet not found: 8b1b17c6705244ecbf38ffd3c5d6d839
W20260807 23:00:09.093024 18525 leader_election.cc:341] T 8b1b17c6705244ecbf38ffd3c5d6d839 P cd086879c54a40bdb4dfecda24a0736b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 102b2e12e0f546e0a4f6402e8bd91481 (127.25.94.2:39593): Not found: Tablet not found: 8b1b17c6705244ecbf38ffd3c5d6d839
I20260807 23:00:09.093127 18525 leader_election.cc:302] T 8b1b17c6705244ecbf38ffd3c5d6d839 P cd086879c54a40bdb4dfecda24a0736b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd086879c54a40bdb4dfecda24a0736b; no voters: 102b2e12e0f546e0a4f6402e8bd91481, 1601d737452f4cb5bd908fbd84c183a9
I20260807 23:00:09.093448 19200 raft_consensus.cc:2726] T 8b1b17c6705244ecbf38ffd3c5d6d839 P cd086879c54a40bdb4dfecda24a0736b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 23:00:09.281828 19201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:00:09.280986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:42307 (local address 127.25.94.2:57043)
0807 23:00:09.281543 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0807 23:00:09.281632 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.62:42307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":339,"threads_started":1}
I20260807 23:00:09.750174 19205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:00:09.749355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:42307 (local address 127.25.94.3:33615)
0807 23:00:09.749880 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0807 23:00:09.749972 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.62:42307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":286,"threads_started":1}
I20260807 23:00:09.810848 19206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:00:09.810038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:42307 (local address 127.25.94.1:37799)
0807 23:00:09.810543 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0807 23:00:09.810636 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.62:42307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":256,"threads_started":1}
I20260807 23:00:10.283735 19209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:00:10.282853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:42307 (local address 127.25.94.2:50589)
0807 23:00:10.283432 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0807 23:00:10.283544 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.62:42307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":284,"threads_started":1}
I20260807 23:00:10.751888 19210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:00:10.751122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:42307 (local address 127.25.94.3:33407)
0807 23:00:10.751622 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0807 23:00:10.751714 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.62:42307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":253,"threads_started":1}
I20260807 23:00:10.812510 19211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:00:10.811765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:42307 (local address 127.25.94.1:43291)
0807 23:00:10.812240 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0807 23:00:10.812341 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.62:42307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":247,"threads_started":1}
I20260807 23:00:11.285847 19214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:00:11.284891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:42307 (local address 127.25.94.2:57295)
0807 23:00:11.285551 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0807 23:00:11.285657 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.62:42307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":378,"threads_started":1}
I20260807 23:00:11.651928 19216 raft_consensus.cc:482] T 8b1b17c6705244ecbf38ffd3c5d6d839 P cd086879c54a40bdb4dfecda24a0736b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 23:00:11.652161 19216 raft_consensus.cc:504] T 8b1b17c6705244ecbf38ffd3c5d6d839 P cd086879c54a40bdb4dfecda24a0736b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd086879c54a40bdb4dfecda24a0736b" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 46643 } } peers { permanent_uuid: "102b2e12e0f546e0a4f6402e8bd91481" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 39593 } } peers { permanent_uuid: "1601d737452f4cb5bd908fbd84c183a9" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 34735 } }
I20260807 23:00:11.652864 19216 leader_election.cc:288] T 8b1b17c6705244ecbf38ffd3c5d6d839 P cd086879c54a40bdb4dfecda24a0736b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 102b2e12e0f546e0a4f6402e8bd91481 (127.25.94.2:39593), 1601d737452f4cb5bd908fbd84c183a9 (127.25.94.3:34735)
W20260807 23:00:11.653954 18523 leader_election.cc:341] T 8b1b17c6705244ecbf38ffd3c5d6d839 P cd086879c54a40bdb4dfecda24a0736b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1601d737452f4cb5bd908fbd84c183a9 (127.25.94.3:34735): Not found: Tablet not found: 8b1b17c6705244ecbf38ffd3c5d6d839
W20260807 23:00:11.654215 18525 leader_election.cc:341] T 8b1b17c6705244ecbf38ffd3c5d6d839 P cd086879c54a40bdb4dfecda24a0736b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 102b2e12e0f546e0a4f6402e8bd91481 (127.25.94.2:39593): Not found: Tablet not found: 8b1b17c6705244ecbf38ffd3c5d6d839
I20260807 23:00:11.654341 18525 leader_election.cc:302] T 8b1b17c6705244ecbf38ffd3c5d6d839 P cd086879c54a40bdb4dfecda24a0736b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd086879c54a40bdb4dfecda24a0736b; no voters: 102b2e12e0f546e0a4f6402e8bd91481, 1601d737452f4cb5bd908fbd84c183a9
I20260807 23:00:11.654641 19216 raft_consensus.cc:2726] T 8b1b17c6705244ecbf38ffd3c5d6d839 P cd086879c54a40bdb4dfecda24a0736b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 23:00:11.753638 19219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:00:11.752832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:42307 (local address 127.25.94.3:45949)
0807 23:00:11.753368 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0807 23:00:11.753453 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.62:42307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":249,"threads_started":1}
I20260807 23:00:11.814460 19220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:00:11.813632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:42307 (local address 127.25.94.1:53035)
0807 23:00:11.814107 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0807 23:00:11.814208 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.62:42307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":232,"threads_started":1}
I20260807 23:00:12.287696 19221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:00:12.286922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:42307 (local address 127.25.94.2:42025)
0807 23:00:12.287432 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0807 23:00:12.287529 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.62:42307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":249,"threads_started":1}
I20260807 23:00:12.755445 19222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:00:12.754642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:42307 (local address 127.25.94.3:44275)
0807 23:00:12.755162 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0807 23:00:12.755263 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.62:42307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":265,"threads_started":1}
I20260807 23:00:12.816334 19223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:00:12.815556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:42307 (local address 127.25.94.1:39629)
0807 23:00:12.816024 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0807 23:00:12.816115 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.62:42307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":259,"threads_started":1}
I20260807 23:00:13.289741 19225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:00:13.288740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:42307 (local address 127.25.94.2:50799)
0807 23:00:13.289341 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0807 23:00:13.289477 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.62:42307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":263,"threads_started":1}
I20260807 23:00:13.757375 19226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:00:13.756425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:42307 (local address 127.25.94.3:38355)
0807 23:00:13.756951 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0807 23:00:13.757037 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.62:42307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":280,"threads_started":1}
I20260807 23:00:13.818495 19227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:00:13.817699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:42307 (local address 127.25.94.1:39455)
0807 23:00:13.818207 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0807 23:00:13.818302 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.62:42307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":252,"threads_started":1}
I20260807 23:00:14.291770 19228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:00:14.290930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:42307 (local address 127.25.94.2:39783)
0807 23:00:14.291465 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0807 23:00:14.291603 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.62:42307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":285,"threads_started":1}
I20260807 23:00:14.738981 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 18443
I20260807 23:00:14.759438 19232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:00:14.758565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:42307 (local address 127.25.94.3:33287)
0807 23:00:14.759141 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0807 23:00:14.759264 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.62:42307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":273,"threads_started":1}
I20260807 23:00:14.760617 19230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:00:14.740147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53956 (local address 127.25.94.1:46643)
0807 23:00:14.740748 (+ 601us) server_negotiation.cc:206] Beginning negotiation
0807 23:00:14.740761 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0807 23:00:14.740800 (+ 39us) server_negotiation.cc:407] Connection header received
0807 23:00:14.740894 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:00:14.740903 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:00:14.741127 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:00:14.741373 (+ 246us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:00:14.743971 (+ 2598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:00:14.744929 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:00:14.746210 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:00:14.746791 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:00:14.746893 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:00:14.754514 (+ 7621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:00:14.754527 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:00:14.754542 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 23:00:14.754584 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:00:14.757699 (+ 3115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:00:14.758432 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:00:14.758442 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:00:14.758445 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:00:14.758626 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:00:14.759120 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:00:14.759124 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:00:14.759126 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:00:14.759517 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:00:14.759831 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0807 23:00:14.760168 (+ 337us) server_negotiation.cc:299] Negotiation successful
0807 23:00:14.760375 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":288,"threads_started":1}
I20260807 23:00:14.762006 18703 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 23:00:14.855829 19234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:00:14.854705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:42307 (local address 127.25.94.1:36539)
0807 23:00:14.855309 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0807 23:00:14.855551 (+ 242us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.62:42307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":281,"threads_started":1}
I20260807 23:00:14.856184 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 18443
I20260807 23:00:14.885253 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 18715
I20260807 23:00:14.899960 19235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:00:14.886141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42430 (local address 127.25.94.2:39593)
0807 23:00:14.886672 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0807 23:00:14.886676 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 23:00:14.886693 (+ 17us) server_negotiation.cc:407] Connection header received
0807 23:00:14.886749 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:00:14.886753 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:00:14.886814 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:00:14.886911 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:00:14.888439 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:00:14.889401 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:00:14.890782 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:00:14.891205 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:00:14.891304 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:00:14.894202 (+ 2898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:00:14.894208 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:00:14.894211 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 23:00:14.894246 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:00:14.897067 (+ 2821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:00:14.897827 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:00:14.897831 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:00:14.897835 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:00:14.897957 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:00:14.898441 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:00:14.898446 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:00:14.898449 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:00:14.898731 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:00:14.899027 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0807 23:00:14.899319 (+ 292us) server_negotiation.cc:299] Negotiation successful
0807 23:00:14.899552 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":310,"threads_started":1}
I20260807 23:00:14.900713 18924 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 23:00:14.986186 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 18715
I20260807 23:00:15.013134 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 18937
I20260807 23:00:15.027942 19237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:00:15.014227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53928 (local address 127.25.94.3:34735)
0807 23:00:15.014877 (+ 650us) server_negotiation.cc:206] Beginning negotiation
0807 23:00:15.014882 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 23:00:15.014899 (+ 17us) server_negotiation.cc:407] Connection header received
0807 23:00:15.014953 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:00:15.014957 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:00:15.015025 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:00:15.015118 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:00:15.016563 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:00:15.017601 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:00:15.018866 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:00:15.019378 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:00:15.019496 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:00:15.022359 (+ 2863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:00:15.022365 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:00:15.022368 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 23:00:15.022406 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:00:15.025139 (+ 2733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:00:15.025922 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:00:15.025930 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:00:15.025932 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:00:15.026098 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:00:15.026620 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:00:15.026624 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:00:15.026626 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:00:15.026928 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:00:15.027165 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0807 23:00:15.027465 (+ 300us) server_negotiation.cc:299] Negotiation successful
0807 23:00:15.027647 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":493,"thread_start_us":313,"threads_started":1}
I20260807 23:00:15.028749 19133 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 23:00:15.117290 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 18937
2026-08-07T23:00:15Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (25696 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 23:00:15 dist-test-slave-z8x9 krb5kdc[19243](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 23:00:15 dist-test-slave-z8x9 krb5kdc[19243](info): set up 2 sockets
Aug 07 23:00:15 dist-test-slave-z8x9 krb5kdc[19243](info): commencing operation
krb5kdc: starting...
W20260807 23:00:16.522923 25976 mini_kdc.cc:121] Time spent starting KDC: real 1.353s user 0.000s sys 0.042s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/krb5kdc/test-user.keytab.
Aug 07 23:00:16 dist-test-slave-z8x9 krb5kdc[19243](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143616, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T23:00:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T23:00:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.94.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.62.keytab.
Entry for principal kudu/127.25.94.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.62.keytab.
I20260807 23:00:16.826479 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.62:40547
--webserver_interface=127.25.94.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.62.keytab
--principal=kudu/127.25.94.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.20:35499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.94.62:40547
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 23:00:17.189723 19261 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 23:00:17.190100 19261 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 23:00:17.190276 19261 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 23:00:17.197981 19261 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260807 23:00:17.198150 19261 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260807 23:00:17.198246 19261 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260807 23:00:17.198356 19261 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 23:00:17.198472 19261 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 23:00:17.198599 19261 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 23:00:17.198693 19261 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 23:00:17.198776 19261 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 23:00:17.209422 19261 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:35499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.25.94.62:40547
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.62.keytab
--principal=kudu/127.25.94.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.62:40547
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.94.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 23:00:17.210878 19261 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 23:00:17.212385 19261 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 23:00:17.225872 19269 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 23:00:17.225817 19267 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 23:00:17.225912 19272 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 23:00:17.226246 19261 server_base.cc:1034] running on GCE node
I20260807 23:00:17.227372 19261 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 23:00:17.228924 19261 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 23:00:17.230182 19261 hybrid_clock.cc:648] HybridClock initialized: now 1786143617230080 us; error 81 us; skew 500 ppm
I20260807 23:00:17.230580 19261 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 23:00:17 dist-test-slave-z8x9 krb5kdc[19243](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143617, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 23:00:17.234133 19261 init.cc:377] Logged in from keytab as kudu/127.25.94.62@KRBTEST.COM (short username kudu)
I20260807 23:00:17.235847 19261 webserver.cc:460] Webserver started at http://127.25.94.62:44809/ using document root <none> and password file <none>
I20260807 23:00:17.236618 19261 fs_manager.cc:359] Metadata directory not provided
I20260807 23:00:17.236764 19261 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 23:00:17.237098 19261 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 23:00:17.240239 19261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/master-0/data/instance:
uuid: "7cf6aa251b54475d819168a882c99955"
format_stamp: "Formatted at 2026-08-07 23:00:17 on dist-test-slave-z8x9"
server_key: "bc426fdb20e3d3a60185290f897b98e4"
server_key_iv: "388cc39b6271fbc10782fbce5d3e08ff"
server_key_version: "encryptionkey@0"
I20260807 23:00:17.241080 19261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/master-0/wal/instance:
uuid: "7cf6aa251b54475d819168a882c99955"
format_stamp: "Formatted at 2026-08-07 23:00:17 on dist-test-slave-z8x9"
server_key: "bc426fdb20e3d3a60185290f897b98e4"
server_key_iv: "388cc39b6271fbc10782fbce5d3e08ff"
server_key_version: "encryptionkey@0"
I20260807 23:00:17.246518 19261 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260807 23:00:17.250330 19283 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 23:00:17.252177 19261 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.004s
I20260807 23:00:17.252396 19261 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/master-0/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/master-0/wal
uuid: "7cf6aa251b54475d819168a882c99955"
format_stamp: "Formatted at 2026-08-07 23:00:17 on dist-test-slave-z8x9"
server_key: "bc426fdb20e3d3a60185290f897b98e4"
server_key_iv: "388cc39b6271fbc10782fbce5d3e08ff"
server_key_version: "encryptionkey@0"
I20260807 23:00:17.252663 19261 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 23:00:17.279197 19261 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 23:00:17.284073 19261 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 23:00:17.284392 19261 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 23:00:17.304584 19261 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.62:40547
I20260807 23:00:17.304610 19376 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.62:40547 every 8 connection(s)
I20260807 23:00:17.306157 19261 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
I20260807 23:00:17.306900 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 19261
I20260807 23:00:17.307150 25976 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/master-0/wal/instance
I20260807 23:00:17.308035 25976 external_mini_cluster.cc:1428] Setting key 966845f10ac9f98c2baf0325a351b2ce
I20260807 23:00:17.311112 19379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 07 23:00:17 dist-test-slave-z8x9 krb5kdc[19243](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143616, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.62@KRBTEST.COM
I20260807 23:00:17.326156 19379 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7cf6aa251b54475d819168a882c99955: Bootstrap starting.
I20260807 23:00:17.330179 19379 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7cf6aa251b54475d819168a882c99955: Neither blocks nor log segments found. Creating new log.
I20260807 23:00:17.331321 19379 log.cc:824] T 00000000000000000000000000000000 P 7cf6aa251b54475d819168a882c99955: Log is configured to *not* fsync() on all Append() calls
I20260807 23:00:17.334445 19382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:00:17.310272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40406 (local address 127.25.94.62:40547)
0807 23:00:17.311092 (+ 820us) server_negotiation.cc:206] Beginning negotiation
0807 23:00:17.311108 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0807 23:00:17.311158 (+ 50us) server_negotiation.cc:407] Connection header received
0807 23:00:17.312866 (+ 1708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:00:17.312889 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:00:17.313345 (+ 456us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:00:17.313697 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:00:17.315725 (+ 2028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:00:17.317339 (+ 1614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:00:17.319636 (+ 2297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:00:17.320464 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:00:17.320624 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:00:17.326164 (+ 5540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:00:17.326175 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:00:17.326199 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 23:00:17.326242 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:00:17.329889 (+ 3647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:00:17.330753 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:00:17.330762 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:00:17.330768 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:00:17.330954 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:00:17.331386 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:00:17.331392 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:00:17.331395 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:00:17.332216 (+ 821us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:00:17.332571 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0807 23:00:17.332955 (+ 384us) server_negotiation.cc:299] Negotiation successful
0807 23:00:17.333434 (+ 479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":614,"thread_start_us":388,"threads_started":1}
I20260807 23:00:17.334780 19379 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7cf6aa251b54475d819168a882c99955: No bootstrap required, opened a new log
I20260807 23:00:17.342306 19379 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7cf6aa251b54475d819168a882c99955 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cf6aa251b54475d819168a882c99955" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 40547 } }
I20260807 23:00:17.342710 19379 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7cf6aa251b54475d819168a882c99955 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 23:00:17.342846 19379 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7cf6aa251b54475d819168a882c99955 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7cf6aa251b54475d819168a882c99955, State: Initialized, Role: FOLLOWER
I20260807 23:00:17.343465 19379 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7cf6aa251b54475d819168a882c99955 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cf6aa251b54475d819168a882c99955" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 40547 } }
I20260807 23:00:17.343720 19379 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7cf6aa251b54475d819168a882c99955 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 23:00:17.343864 19379 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7cf6aa251b54475d819168a882c99955 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 23:00:17.344054 19379 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7cf6aa251b54475d819168a882c99955 [term 0 FOLLOWER]: Advancing to term 1
I20260807 23:00:17.346343 19379 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7cf6aa251b54475d819168a882c99955 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cf6aa251b54475d819168a882c99955" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 40547 } }
I20260807 23:00:17.346841 19379 leader_election.cc:302] T 00000000000000000000000000000000 P 7cf6aa251b54475d819168a882c99955 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7cf6aa251b54475d819168a882c99955; no voters:
I20260807 23:00:17.347424 19379 leader_election.cc:288] T 00000000000000000000000000000000 P 7cf6aa251b54475d819168a882c99955 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 23:00:17.347581 19385 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7cf6aa251b54475d819168a882c99955 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 23:00:17.348291 19385 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7cf6aa251b54475d819168a882c99955 [term 1 LEADER]: Becoming Leader. State: Replica: 7cf6aa251b54475d819168a882c99955, State: Running, Role: LEADER
I20260807 23:00:17.348944 19385 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7cf6aa251b54475d819168a882c99955 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cf6aa251b54475d819168a882c99955" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 40547 } }
I20260807 23:00:17.349615 19379 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7cf6aa251b54475d819168a882c99955 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 23:00:17.351536 19386 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7cf6aa251b54475d819168a882c99955 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7cf6aa251b54475d819168a882c99955" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cf6aa251b54475d819168a882c99955" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 40547 } } }
I20260807 23:00:17.351883 19386 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7cf6aa251b54475d819168a882c99955 [sys.catalog]: This master's current role is: LEADER
I20260807 23:00:17.352813 19395 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 23:00:17.352670 19388 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7cf6aa251b54475d819168a882c99955 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7cf6aa251b54475d819168a882c99955. Latest consensus state: current_term: 1 leader_uuid: "7cf6aa251b54475d819168a882c99955" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cf6aa251b54475d819168a882c99955" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 40547 } } }
I20260807 23:00:17.355577 19388 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7cf6aa251b54475d819168a882c99955 [sys.catalog]: This master's current role is: LEADER
I20260807 23:00:17.363592 19395 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 23:00:17.373782 19395 catalog_manager.cc:1347] Generated new cluster ID: 7a21efc46e6c4d0689df4ef042c69b80
I20260807 23:00:17.373958 19395 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 23:00:17.391901 19395 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 23:00:17.393955 19395 catalog_manager.cc:1504] Loading token signing keys...
I20260807 23:00:17.407979 19395 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7cf6aa251b54475d819168a882c99955: Generated new TSK 0
I20260807 23:00:17.409078 19395 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.94.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.1.keytab.
Entry for principal kudu/127.25.94.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.1.keytab.
I20260807 23:00:17.485044 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.1:0
--local_ip_for_outbound_sockets=127.25.94.1
--webserver_interface=127.25.94.1
--webserver_port=0
--tserver_master_addrs=127.25.94.62:40547
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.1.keytab
--principal=kudu/127.25.94.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.20:35499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 23:00:17.840050 19418 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 23:00:17.840549 19418 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 23:00:17.840745 19418 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 23:00:17.854424 19418 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 23:00:17.854592 19418 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 23:00:17.854807 19418 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.1
I20260807 23:00:17.873776 19418 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:35499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.1.keytab
--principal=kudu/127.25.94.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.94.1
--webserver_port=0
--tserver_master_addrs=127.25.94.62:40547
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.1
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 23:00:17.875599 19418 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 23:00:17.877417 19418 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 23:00:17.893606 19425 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 23:00:17.893553 19426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 23:00:17.894263 19429 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 23:00:17.894912 19418 server_base.cc:1034] running on GCE node
I20260807 23:00:17.895787 19418 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 23:00:17.897109 19418 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 23:00:17.898442 19418 hybrid_clock.cc:648] HybridClock initialized: now 1786143617898315 us; error 106 us; skew 500 ppm
I20260807 23:00:17.898910 19418 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 23:00:17 dist-test-slave-z8x9 krb5kdc[19243](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143617, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 23:00:17.903440 19418 init.cc:377] Logged in from keytab as kudu/127.25.94.1@KRBTEST.COM (short username kudu)
I20260807 23:00:17.904994 19418 webserver.cc:460] Webserver started at http://127.25.94.1:35209/ using document root <none> and password file <none>
I20260807 23:00:17.905829 19418 fs_manager.cc:359] Metadata directory not provided
I20260807 23:00:17.905949 19418 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 23:00:17.906220 19418 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 23:00:17.909495 19418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-0/data/instance:
uuid: "8a9f21049c9c4db48d48db896dcce03d"
format_stamp: "Formatted at 2026-08-07 23:00:17 on dist-test-slave-z8x9"
server_key: "fe42c621df0923528e732238bb21d9b0"
server_key_iv: "3420d0b1ab3d2891a30b640606475760"
server_key_version: "encryptionkey@0"
I20260807 23:00:17.910221 19418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-0/wal/instance:
uuid: "8a9f21049c9c4db48d48db896dcce03d"
format_stamp: "Formatted at 2026-08-07 23:00:17 on dist-test-slave-z8x9"
server_key: "fe42c621df0923528e732238bb21d9b0"
server_key_iv: "3420d0b1ab3d2891a30b640606475760"
server_key_version: "encryptionkey@0"
I20260807 23:00:17.915501 19418 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260807 23:00:17.918850 19443 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 23:00:17.920166 19418 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260807 23:00:17.920306 19418 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-0/wal
uuid: "8a9f21049c9c4db48d48db896dcce03d"
format_stamp: "Formatted at 2026-08-07 23:00:17 on dist-test-slave-z8x9"
server_key: "fe42c621df0923528e732238bb21d9b0"
server_key_iv: "3420d0b1ab3d2891a30b640606475760"
server_key_version: "encryptionkey@0"
I20260807 23:00:17.920464 19418 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 23:00:17.933607 19418 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 23:00:17.938225 19418 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 23:00:17.938484 19418 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 23:00:17.939401 19418 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 23:00:17.940765 19418 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 23:00:17.940855 19418 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 23:00:17.940943 19418 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 23:00:17.940982 19418 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 23:00:17.975241 19418 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.1:45459
I20260807 23:00:17.975286 19626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.1:45459 every 8 connection(s)
I20260807 23:00:17.976643 19418 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-0/data/info.pb
I20260807 23:00:17.986415 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 19418
I20260807 23:00:17.986677 25976 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-0/wal/instance
I20260807 23:00:17.987425 25976 external_mini_cluster.cc:1428] Setting key d468ec0bf5230978a4590812910bf39a
Aug 07 23:00:17 dist-test-slave-z8x9 krb5kdc[19243](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143617, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.1@KRBTEST.COM for kudu/127.25.94.62@KRBTEST.COM
I20260807 23:00:17.999827 19632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:00:17.979457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.1:50965 (local address 127.25.94.62:40547)
0807 23:00:17.979957 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0807 23:00:17.979965 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 23:00:17.980986 (+ 1021us) server_negotiation.cc:407] Connection header received
0807 23:00:17.982877 (+ 1891us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:00:17.982882 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:00:17.982965 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:00:17.983066 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:00:17.984616 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:00:17.985501 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:00:17.987386 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:00:17.988008 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:00:17.988163 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:00:17.992486 (+ 4323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:00:17.992495 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:00:17.992503 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 23:00:17.992550 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:00:17.996306 (+ 3756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:00:17.997117 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:00:17.997125 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:00:17.997131 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:00:17.997337 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:00:17.997773 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:00:17.997781 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:00:17.997788 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:00:17.998184 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:00:17.998334 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0807 23:00:17.999046 (+ 712us) server_negotiation.cc:299] Negotiation successful
0807 23:00:17.999305 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":275,"threads_started":1}
I20260807 23:00:18.000466 19636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:00:17.980062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:40547 (local address 127.25.94.1:50965)
0807 23:00:17.980805 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0807 23:00:17.980850 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0807 23:00:17.982599 (+ 1749us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 23:00:17.983275 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 23:00:17.983283 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 23:00:17.983822 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0807 23:00:17.984399 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:00:17.984415 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:00:17.985714 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:00:17.985718 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:00:17.987216 (+ 1498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:00:17.987230 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:00:17.988295 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:00:17.988304 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:00:17.988624 (+ 320us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:00:17.989514 (+ 890us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 23:00:17.989538 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0807 23:00:17.992242 (+ 2704us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 23:00:17.996538 (+ 4296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 23:00:17.996545 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 23:00:17.996559 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0807 23:00:17.996980 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 23:00:17.997452 (+ 472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 23:00:17.997455 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 23:00:17.997457 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 23:00:17.997652 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 23:00:17.998315 (+ 663us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 23:00:17.998322 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 23:00:17.998651 (+ 329us) client_negotiation.cc:769] Sending connection context
0807 23:00:17.998881 (+ 230us) client_negotiation.cc:240] Negotiation successful
0807 23:00:17.999275 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":297,"threads_started":1}
I20260807 23:00:18.002147 19630 heartbeater.cc:347] Connected to a master server at 127.25.94.62:40547
I20260807 23:00:18.002535 19630 heartbeater.cc:464] Registering TS with master...
I20260807 23:00:18.003509 19630 heartbeater.cc:511] Master 127.25.94.62:40547 requested a full tablet report, sending...
I20260807 23:00:18.005997 19319 ts_manager.cc:194] Registered new tserver with Master: 8a9f21049c9c4db48d48db896dcce03d (127.25.94.1:45459)
I20260807 23:00:18.007485 19319 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.94.1@KRBTEST.COM'} at 127.25.94.1:50965
WARNING: no policy specified for kudu/127.25.94.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.2.keytab.
Entry for principal kudu/127.25.94.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.2.keytab.
I20260807 23:00:18.056843 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.2:0
--local_ip_for_outbound_sockets=127.25.94.2
--webserver_interface=127.25.94.2
--webserver_port=0
--tserver_master_addrs=127.25.94.62:40547
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.2.keytab
--principal=kudu/127.25.94.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.20:35499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 23:00:18.418815 19641 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 23:00:18.419126 19641 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 23:00:18.419296 19641 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 23:00:18.427853 19641 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 23:00:18.427992 19641 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 23:00:18.428184 19641 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.2
I20260807 23:00:18.442888 19641 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:35499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.2.keytab
--principal=kudu/127.25.94.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.94.2
--webserver_port=0
--tserver_master_addrs=127.25.94.62:40547
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.2
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 23:00:18.444692 19641 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 23:00:18.446127 19641 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 23:00:18.463995 19649 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 23:00:18.464946 19655 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 23:00:18.465372 19648 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 23:00:18.467360 19641 server_base.cc:1034] running on GCE node
I20260807 23:00:18.468318 19641 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 23:00:18.469522 19641 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 23:00:18.470795 19641 hybrid_clock.cc:648] HybridClock initialized: now 1786143618470720 us; error 71 us; skew 500 ppm
I20260807 23:00:18.471177 19641 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 23:00:18 dist-test-slave-z8x9 krb5kdc[19243](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143618, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 23:00:18.474720 19641 init.cc:377] Logged in from keytab as kudu/127.25.94.2@KRBTEST.COM (short username kudu)
I20260807 23:00:18.476583 19641 webserver.cc:460] Webserver started at http://127.25.94.2:37971/ using document root <none> and password file <none>
I20260807 23:00:18.477416 19641 fs_manager.cc:359] Metadata directory not provided
I20260807 23:00:18.477551 19641 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 23:00:18.477873 19641 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 23:00:18.481019 19641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-1/data/instance:
uuid: "490f1fd977fb4fd586aed08107d9f266"
format_stamp: "Formatted at 2026-08-07 23:00:18 on dist-test-slave-z8x9"
server_key: "ad85a15d8b332a3dd74c4377ce274f8a"
server_key_iv: "6ea30c2da1c71d3dd026ef2728b7c3bb"
server_key_version: "encryptionkey@0"
I20260807 23:00:18.481880 19641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-1/wal/instance:
uuid: "490f1fd977fb4fd586aed08107d9f266"
format_stamp: "Formatted at 2026-08-07 23:00:18 on dist-test-slave-z8x9"
server_key: "ad85a15d8b332a3dd74c4377ce274f8a"
server_key_iv: "6ea30c2da1c71d3dd026ef2728b7c3bb"
server_key_version: "encryptionkey@0"
I20260807 23:00:18.487646 19641 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.002s
I20260807 23:00:18.491411 19666 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 23:00:18.492884 19641 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 23:00:18.493062 19641 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-1/wal
uuid: "490f1fd977fb4fd586aed08107d9f266"
format_stamp: "Formatted at 2026-08-07 23:00:18 on dist-test-slave-z8x9"
server_key: "ad85a15d8b332a3dd74c4377ce274f8a"
server_key_iv: "6ea30c2da1c71d3dd026ef2728b7c3bb"
server_key_version: "encryptionkey@0"
I20260807 23:00:18.493341 19641 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 23:00:18.510635 19641 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 23:00:18.515156 19641 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 23:00:18.515458 19641 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 23:00:18.516557 19641 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 23:00:18.518090 19641 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 23:00:18.518163 19641 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 23:00:18.518321 19641 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 23:00:18.518373 19641 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 23:00:18.562318 19641 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.2:36939
I20260807 23:00:18.562345 19866 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.2:36939 every 8 connection(s)
I20260807 23:00:18.563663 19641 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-1/data/info.pb
I20260807 23:00:18.564414 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 19641
I20260807 23:00:18.564792 25976 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-1/wal/instance
I20260807 23:00:18.565483 25976 external_mini_cluster.cc:1428] Setting key 87af8b77a1190017fd66695de40d65a0
Aug 07 23:00:18 dist-test-slave-z8x9 krb5kdc[19243](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143618, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.2@KRBTEST.COM for kudu/127.25.94.62@KRBTEST.COM
I20260807 23:00:18.601395 19871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:00:18.567931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.2:41653 (local address 127.25.94.62:40547)
0807 23:00:18.581628 (+ 13697us) server_negotiation.cc:206] Beginning negotiation
0807 23:00:18.581635 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 23:00:18.581657 (+ 22us) server_negotiation.cc:407] Connection header received
0807 23:00:18.581741 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:00:18.581746 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:00:18.581847 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:00:18.581965 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:00:18.583807 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:00:18.584680 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:00:18.586837 (+ 2157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:00:18.587548 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:00:18.587688 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:00:18.592575 (+ 4887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:00:18.592584 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:00:18.592592 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 23:00:18.592643 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:00:18.596745 (+ 4102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:00:18.597975 (+ 1230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:00:18.597983 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:00:18.597990 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:00:18.598264 (+ 274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:00:18.598851 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:00:18.598860 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:00:18.598865 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:00:18.599316 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:00:18.599477 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0807 23:00:18.600554 (+ 1077us) server_negotiation.cc:299] Negotiation successful
0807 23:00:18.600843 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13560,"thread_start_us":13443,"threads_started":1}
I20260807 23:00:18.602731 19872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:00:18.567925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:40547 (local address 127.25.94.2:41653)
0807 23:00:18.578806 (+ 10881us) negotiation.cc:107] Waiting for socket to connect
0807 23:00:18.578854 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0807 23:00:18.580675 (+ 1821us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 23:00:18.582341 (+ 1666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 23:00:18.582350 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 23:00:18.582933 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0807 23:00:18.583587 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:00:18.583607 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:00:18.584951 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:00:18.584958 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:00:18.586655 (+ 1697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:00:18.586669 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:00:18.588052 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:00:18.588061 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:00:18.588407 (+ 346us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:00:18.589292 (+ 885us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 23:00:18.589326 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0807 23:00:18.592226 (+ 2900us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 23:00:18.596919 (+ 4693us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 23:00:18.596932 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 23:00:18.596956 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0807 23:00:18.597814 (+ 858us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 23:00:18.598392 (+ 578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 23:00:18.598400 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 23:00:18.598405 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 23:00:18.598688 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 23:00:18.599462 (+ 774us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 23:00:18.599474 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 23:00:18.600012 (+ 538us) client_negotiation.cc:769] Sending connection context
0807 23:00:18.600402 (+ 390us) client_negotiation.cc:240] Negotiation successful
0807 23:00:18.601095 (+ 693us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10573,"thread_start_us":13009,"threads_started":1}
I20260807 23:00:18.604518 19868 heartbeater.cc:347] Connected to a master server at 127.25.94.62:40547
I20260807 23:00:18.604951 19868 heartbeater.cc:464] Registering TS with master...
I20260807 23:00:18.606000 19868 heartbeater.cc:511] Master 127.25.94.62:40547 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.25.94.3@KRBTEST.COM; defaulting to no policy
I20260807 23:00:18.607865 19319 ts_manager.cc:194] Registered new tserver with Master: 490f1fd977fb4fd586aed08107d9f266 (127.25.94.2:36939)
I20260807 23:00:18.609330 19319 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.94.2@KRBTEST.COM'} at 127.25.94.2:41653
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.3.keytab.
Entry for principal kudu/127.25.94.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.3.keytab.
I20260807 23:00:18.644737 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.3:0
--local_ip_for_outbound_sockets=127.25.94.3
--webserver_interface=127.25.94.3
--webserver_port=0
--tserver_master_addrs=127.25.94.62:40547
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.3.keytab
--principal=kudu/127.25.94.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.20:35499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 23:00:19.011344 19630 heartbeater.cc:503] Master 127.25.94.62:40547 was elected leader, sending a full tablet report...
W20260807 23:00:19.013670 19876 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 23:00:19.014009 19876 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 23:00:19.014186 19876 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 23:00:19.021889 19876 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 23:00:19.022045 19876 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 23:00:19.022223 19876 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.3
I20260807 23:00:19.032738 19876 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:35499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.3.keytab
--principal=kudu/127.25.94.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.94.3
--webserver_port=0
--tserver_master_addrs=127.25.94.62:40547
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.3
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 23:00:19.034170 19876 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 23:00:19.035497 19876 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 23:00:19.049105 19882 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 23:00:19.049052 19883 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 23:00:19.049022 19888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 23:00:19.049888 19876 server_base.cc:1034] running on GCE node
I20260807 23:00:19.050592 19876 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 23:00:19.051582 19876 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 23:00:19.052830 19876 hybrid_clock.cc:648] HybridClock initialized: now 1786143619052783 us; error 56 us; skew 500 ppm
I20260807 23:00:19.053207 19876 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 23:00:19 dist-test-slave-z8x9 krb5kdc[19243](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143619, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 23:00:19.056735 19876 init.cc:377] Logged in from keytab as kudu/127.25.94.3@KRBTEST.COM (short username kudu)
I20260807 23:00:19.058509 19876 webserver.cc:460] Webserver started at http://127.25.94.3:33175/ using document root <none> and password file <none>
I20260807 23:00:19.059284 19876 fs_manager.cc:359] Metadata directory not provided
I20260807 23:00:19.059456 19876 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 23:00:19.059887 19876 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 23:00:19.063263 19876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-2/data/instance:
uuid: "a49447d7b0024e12b538ea065f867cb5"
format_stamp: "Formatted at 2026-08-07 23:00:19 on dist-test-slave-z8x9"
server_key: "104d48c30a3e317d64e2257306c7c74f"
server_key_iv: "a256b9d5ac16994f5bb2160b184176ec"
server_key_version: "encryptionkey@0"
I20260807 23:00:19.064066 19876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-2/wal/instance:
uuid: "a49447d7b0024e12b538ea065f867cb5"
format_stamp: "Formatted at 2026-08-07 23:00:19 on dist-test-slave-z8x9"
server_key: "104d48c30a3e317d64e2257306c7c74f"
server_key_iv: "a256b9d5ac16994f5bb2160b184176ec"
server_key_version: "encryptionkey@0"
I20260807 23:00:19.069721 19876 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260807 23:00:19.073506 19897 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 23:00:19.074865 19876 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 23:00:19.075070 19876 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-2/wal
uuid: "a49447d7b0024e12b538ea065f867cb5"
format_stamp: "Formatted at 2026-08-07 23:00:19 on dist-test-slave-z8x9"
server_key: "104d48c30a3e317d64e2257306c7c74f"
server_key_iv: "a256b9d5ac16994f5bb2160b184176ec"
server_key_version: "encryptionkey@0"
I20260807 23:00:19.075284 19876 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 23:00:19.095956 19876 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 23:00:19.101162 19876 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 23:00:19.101547 19876 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 23:00:19.102818 19876 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 23:00:19.104570 19876 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 23:00:19.104719 19876 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 23:00:19.104897 19876 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 23:00:19.104975 19876 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 23:00:19.145155 19876 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.3:41199
I20260807 23:00:19.145304 20082 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.3:41199 every 8 connection(s)
I20260807 23:00:19.146895 19876 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-2/data/info.pb
I20260807 23:00:19.151409 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 19876
I20260807 23:00:19.151641 25976 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-2/wal/instance
I20260807 23:00:19.152308 25976 external_mini_cluster.cc:1428] Setting key 3a6762e920141b574ec80f592ceded65
Aug 07 23:00:19 dist-test-slave-z8x9 krb5kdc[19243](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143619, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.3@KRBTEST.COM for kudu/127.25.94.62@KRBTEST.COM
I20260807 23:00:19.172205 20087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:00:19.149872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.3:55903 (local address 127.25.94.62:40547)
0807 23:00:19.150476 (+ 604us) server_negotiation.cc:206] Beginning negotiation
0807 23:00:19.150481 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 23:00:19.152994 (+ 2513us) server_negotiation.cc:407] Connection header received
0807 23:00:19.154340 (+ 1346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:00:19.154345 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:00:19.154444 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:00:19.154606 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:00:19.156846 (+ 2240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:00:19.157834 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:00:19.159688 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:00:19.160349 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:00:19.160457 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:00:19.164838 (+ 4381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:00:19.164845 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:00:19.164848 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 23:00:19.164887 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:00:19.168346 (+ 3459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:00:19.169409 (+ 1063us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:00:19.169414 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:00:19.169416 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:00:19.169602 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:00:19.170183 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:00:19.170188 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:00:19.170190 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:00:19.170510 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:00:19.170659 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0807 23:00:19.171551 (+ 892us) server_negotiation.cc:299] Negotiation successful
0807 23:00:19.171879 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":472,"thread_start_us":236,"threads_started":1}
I20260807 23:00:19.173063 20088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:00:19.150551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:40547 (local address 127.25.94.3:55903)
0807 23:00:19.151531 (+ 980us) negotiation.cc:107] Waiting for socket to connect
0807 23:00:19.151589 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0807 23:00:19.153918 (+ 2329us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 23:00:19.155073 (+ 1155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 23:00:19.155084 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 23:00:19.155702 (+ 618us) client_negotiation.cc:189] Negotiated authn=SASL
0807 23:00:19.156652 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:00:19.156675 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:00:19.158016 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:00:19.158020 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:00:19.159500 (+ 1480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:00:19.159515 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:00:19.160590 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:00:19.160596 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:00:19.160809 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:00:19.161766 (+ 957us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 23:00:19.161801 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0807 23:00:19.164549 (+ 2748us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 23:00:19.168515 (+ 3966us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 23:00:19.168526 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 23:00:19.168547 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0807 23:00:19.169264 (+ 717us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 23:00:19.169770 (+ 506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 23:00:19.169774 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 23:00:19.169777 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 23:00:19.169994 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 23:00:19.170687 (+ 693us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 23:00:19.170695 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 23:00:19.171097 (+ 402us) client_negotiation.cc:769] Sending connection context
0807 23:00:19.171384 (+ 287us) client_negotiation.cc:240] Negotiation successful
0807 23:00:19.171835 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":639,"thread_start_us":382,"threads_started":1}
I20260807 23:00:19.175043 20083 heartbeater.cc:347] Connected to a master server at 127.25.94.62:40547
I20260807 23:00:19.175367 20083 heartbeater.cc:464] Registering TS with master...
I20260807 23:00:19.176147 20083 heartbeater.cc:511] Master 127.25.94.62:40547 requested a full tablet report, sending...
I20260807 23:00:19.177947 19319 ts_manager.cc:194] Registered new tserver with Master: a49447d7b0024e12b538ea065f867cb5 (127.25.94.3:41199)
I20260807 23:00:19.178817 19319 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.94.3@KRBTEST.COM'} at 127.25.94.3:55903
I20260807 23:00:19.182080 25976 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 23:00:19.202391 20087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:00:19.187209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40416 (local address 127.25.94.62:40547)
0807 23:00:19.187394 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0807 23:00:19.187399 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 23:00:19.187949 (+ 550us) server_negotiation.cc:407] Connection header received
0807 23:00:19.188094 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:00:19.188098 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:00:19.188178 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:00:19.188277 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:00:19.190064 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:00:19.190830 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:00:19.192362 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:00:19.192795 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:00:19.192932 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:00:19.196241 (+ 3309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:00:19.196249 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:00:19.196253 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 23:00:19.196307 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:00:19.199544 (+ 3237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:00:19.200367 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:00:19.200371 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:00:19.200374 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:00:19.200513 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:00:19.200901 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:00:19.200906 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:00:19.200908 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:00:19.201377 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:00:19.201608 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0807 23:00:19.201948 (+ 340us) server_negotiation.cc:299] Negotiation successful
0807 23:00:19.202130 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260807 23:00:19.207171 19319 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40416:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 23:00:19.214658 19319 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 23:00:19.251081 20109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:00:19.229488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48634 (local address 127.25.94.2:36939)
0807 23:00:19.241673 (+ 12185us) server_negotiation.cc:206] Beginning negotiation
0807 23:00:19.241695 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0807 23:00:19.241739 (+ 44us) server_negotiation.cc:407] Connection header received
0807 23:00:19.241888 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:00:19.241901 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:00:19.242285 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:00:19.243562 (+ 1277us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 23:00:19.243840 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:00:19.246045 (+ 2205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:00:19.248362 (+ 2317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:00:19.249833 (+ 1471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:00:19.249971 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:00:19.250172 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0807 23:00:19.250548 (+ 376us) server_negotiation.cc:299] Negotiation successful
0807 23:00:19.250757 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11972,"thread_start_us":11847,"threads_started":1}
I20260807 23:00:19.251204 20106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:00:19.228422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.2:36939 (local address 127.0.0.1:48634)
0807 23:00:19.229273 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0807 23:00:19.229322 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0807 23:00:19.229515 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 23:00:19.242661 (+ 13146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 23:00:19.242673 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 23:00:19.242732 (+ 59us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 23:00:19.243244 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:00:19.243263 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:00:19.246184 (+ 2921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:00:19.246191 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:00:19.248186 (+ 1995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:00:19.248201 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:00:19.250037 (+ 1836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:00:19.250043 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:00:19.250265 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:00:19.250327 (+ 62us) client_negotiation.cc:769] Sending connection context
0807 23:00:19.250470 (+ 143us) client_negotiation.cc:240] Negotiation successful
0807 23:00:19.250718 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"mutex_wait_us":44,"thread_start_us":403,"threads_started":1}
I20260807 23:00:19.251317 20108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:00:19.228997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.1:45459 (local address 127.0.0.1:60160)
0807 23:00:19.242043 (+ 13046us) negotiation.cc:107] Waiting for socket to connect
0807 23:00:19.242066 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0807 23:00:19.242230 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 23:00:19.243252 (+ 1022us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 23:00:19.243257 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 23:00:19.243284 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 23:00:19.243793 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:00:19.243809 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:00:19.245694 (+ 1885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:00:19.245699 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:00:19.248026 (+ 2327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:00:19.248042 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:00:19.250337 (+ 2295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:00:19.250342 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:00:19.250618 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:00:19.250650 (+ 32us) client_negotiation.cc:769] Sending connection context
0807 23:00:19.250739 (+ 89us) client_negotiation.cc:240] Negotiation successful
0807 23:00:19.250866 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12912,"mutex_wait_us":61,"thread_start_us":12765,"threads_started":1}
I20260807 23:00:19.252080 20111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:00:19.229493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60160 (local address 127.25.94.1:45459)
0807 23:00:19.241112 (+ 11619us) server_negotiation.cc:206] Beginning negotiation
0807 23:00:19.241127 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0807 23:00:19.242371 (+ 1244us) server_negotiation.cc:407] Connection header received
0807 23:00:19.242490 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:00:19.242498 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:00:19.242743 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:00:19.242959 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 23:00:19.243984 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:00:19.245531 (+ 1547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:00:19.248200 (+ 2669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:00:19.249833 (+ 1633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:00:19.249971 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:00:19.250183 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0807 23:00:19.251513 (+ 1330us) server_negotiation.cc:299] Negotiation successful
0807 23:00:19.251694 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11343,"thread_start_us":11116,"threads_started":1}
I20260807 23:00:19.253346 20112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:00:19.229572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32950 (local address 127.25.94.3:41199)
0807 23:00:19.244032 (+ 14460us) server_negotiation.cc:206] Beginning negotiation
0807 23:00:19.244038 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 23:00:19.244061 (+ 23us) server_negotiation.cc:407] Connection header received
0807 23:00:19.244130 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:00:19.244135 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:00:19.244356 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:00:19.244576 (+ 220us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 23:00:19.245567 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:00:19.247232 (+ 1665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:00:19.249796 (+ 2564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:00:19.251459 (+ 1663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:00:19.251584 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:00:19.252636 (+ 1052us) server_negotiation.cc:1035] Waiting for connection context
0807 23:00:19.252740 (+ 104us) server_negotiation.cc:299] Negotiation successful
0807 23:00:19.252911 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14287,"thread_start_us":14089,"threads_started":1}
I20260807 23:00:19.254071 20107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:00:19.228634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.3:41199 (local address 127.0.0.1:32950)
0807 23:00:19.239778 (+ 11144us) negotiation.cc:107] Waiting for socket to connect
0807 23:00:19.239807 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0807 23:00:19.240027 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 23:00:19.244884 (+ 4857us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 23:00:19.244889 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 23:00:19.244917 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 23:00:19.245363 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:00:19.245374 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:00:19.247398 (+ 2024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:00:19.247404 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:00:19.249648 (+ 2244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:00:19.249659 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:00:19.251889 (+ 2230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:00:19.251896 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:00:19.252075 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:00:19.252119 (+ 44us) client_negotiation.cc:769] Sending connection context
0807 23:00:19.252332 (+ 213us) client_negotiation.cc:240] Negotiation successful
0807 23:00:19.252488 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10910,"thread_start_us":11530,"threads_started":1}
I20260807 23:00:19.256244 19755 tablet_service.cc:1467] Processing CreateTablet for tablet 76db18bb9aa3487796175ac7face46a9 (DEFAULT_TABLE table=test-table [id=7284e103ec9a4959ad52fd136ba85b91]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 23:00:19.256253 19526 tablet_service.cc:1467] Processing CreateTablet for tablet 76db18bb9aa3487796175ac7face46a9 (DEFAULT_TABLE table=test-table [id=7284e103ec9a4959ad52fd136ba85b91]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 23:00:19.257848 19755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76db18bb9aa3487796175ac7face46a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 23:00:19.257848 19526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76db18bb9aa3487796175ac7face46a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 23:00:19.257478 19979 tablet_service.cc:1467] Processing CreateTablet for tablet 76db18bb9aa3487796175ac7face46a9 (DEFAULT_TABLE table=test-table [id=7284e103ec9a4959ad52fd136ba85b91]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 23:00:19.259557 19979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76db18bb9aa3487796175ac7face46a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 23:00:19.271876 20113 tablet_bootstrap.cc:492] T 76db18bb9aa3487796175ac7face46a9 P a49447d7b0024e12b538ea065f867cb5: Bootstrap starting.
I20260807 23:00:19.274216 20115 tablet_bootstrap.cc:492] T 76db18bb9aa3487796175ac7face46a9 P 8a9f21049c9c4db48d48db896dcce03d: Bootstrap starting.
I20260807 23:00:19.275933 20117 tablet_bootstrap.cc:492] T 76db18bb9aa3487796175ac7face46a9 P 490f1fd977fb4fd586aed08107d9f266: Bootstrap starting.
I20260807 23:00:19.277566 20113 tablet_bootstrap.cc:654] T 76db18bb9aa3487796175ac7face46a9 P a49447d7b0024e12b538ea065f867cb5: Neither blocks nor log segments found. Creating new log.
I20260807 23:00:19.278167 20115 tablet_bootstrap.cc:654] T 76db18bb9aa3487796175ac7face46a9 P 8a9f21049c9c4db48d48db896dcce03d: Neither blocks nor log segments found. Creating new log.
I20260807 23:00:19.278983 20117 tablet_bootstrap.cc:654] T 76db18bb9aa3487796175ac7face46a9 P 490f1fd977fb4fd586aed08107d9f266: Neither blocks nor log segments found. Creating new log.
I20260807 23:00:19.279026 20113 log.cc:824] T 76db18bb9aa3487796175ac7face46a9 P a49447d7b0024e12b538ea065f867cb5: Log is configured to *not* fsync() on all Append() calls
I20260807 23:00:19.279541 20115 log.cc:824] T 76db18bb9aa3487796175ac7face46a9 P 8a9f21049c9c4db48d48db896dcce03d: Log is configured to *not* fsync() on all Append() calls
I20260807 23:00:19.280191 20117 log.cc:824] T 76db18bb9aa3487796175ac7face46a9 P 490f1fd977fb4fd586aed08107d9f266: Log is configured to *not* fsync() on all Append() calls
I20260807 23:00:19.283015 20115 tablet_bootstrap.cc:492] T 76db18bb9aa3487796175ac7face46a9 P 8a9f21049c9c4db48d48db896dcce03d: No bootstrap required, opened a new log
I20260807 23:00:19.283048 20113 tablet_bootstrap.cc:492] T 76db18bb9aa3487796175ac7face46a9 P a49447d7b0024e12b538ea065f867cb5: No bootstrap required, opened a new log
I20260807 23:00:19.283349 20115 ts_tablet_manager.cc:1397] T 76db18bb9aa3487796175ac7face46a9 P 8a9f21049c9c4db48d48db896dcce03d: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.004s
I20260807 23:00:19.283360 20113 ts_tablet_manager.cc:1397] T 76db18bb9aa3487796175ac7face46a9 P a49447d7b0024e12b538ea065f867cb5: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.000s
I20260807 23:00:19.283797 20117 tablet_bootstrap.cc:492] T 76db18bb9aa3487796175ac7face46a9 P 490f1fd977fb4fd586aed08107d9f266: No bootstrap required, opened a new log
I20260807 23:00:19.284067 20117 ts_tablet_manager.cc:1397] T 76db18bb9aa3487796175ac7face46a9 P 490f1fd977fb4fd586aed08107d9f266: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260807 23:00:19.290874 20115 raft_consensus.cc:348] T 76db18bb9aa3487796175ac7face46a9 P 8a9f21049c9c4db48d48db896dcce03d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "490f1fd977fb4fd586aed08107d9f266" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 36939 } } peers { permanent_uuid: "a49447d7b0024e12b538ea065f867cb5" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 41199 } } peers { permanent_uuid: "8a9f21049c9c4db48d48db896dcce03d" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 45459 } }
I20260807 23:00:19.291254 20115 raft_consensus.cc:374] T 76db18bb9aa3487796175ac7face46a9 P 8a9f21049c9c4db48d48db896dcce03d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 23:00:19.291350 20115 raft_consensus.cc:729] T 76db18bb9aa3487796175ac7face46a9 P 8a9f21049c9c4db48d48db896dcce03d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9f21049c9c4db48d48db896dcce03d, State: Initialized, Role: FOLLOWER
I20260807 23:00:19.291967 20115 consensus_queue.cc:260] T 76db18bb9aa3487796175ac7face46a9 P 8a9f21049c9c4db48d48db896dcce03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "490f1fd977fb4fd586aed08107d9f266" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 36939 } } peers { permanent_uuid: "a49447d7b0024e12b538ea065f867cb5" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 41199 } } peers { permanent_uuid: "8a9f21049c9c4db48d48db896dcce03d" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 45459 } }
I20260807 23:00:19.293820 20115 ts_tablet_manager.cc:1428] T 76db18bb9aa3487796175ac7face46a9 P 8a9f21049c9c4db48d48db896dcce03d: Time spent starting tablet: real 0.010s user 0.012s sys 0.000s
I20260807 23:00:19.294215 20113 raft_consensus.cc:348] T 76db18bb9aa3487796175ac7face46a9 P a49447d7b0024e12b538ea065f867cb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "490f1fd977fb4fd586aed08107d9f266" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 36939 } } peers { permanent_uuid: "a49447d7b0024e12b538ea065f867cb5" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 41199 } } peers { permanent_uuid: "8a9f21049c9c4db48d48db896dcce03d" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 45459 } }
I20260807 23:00:19.294663 20113 raft_consensus.cc:374] T 76db18bb9aa3487796175ac7face46a9 P a49447d7b0024e12b538ea065f867cb5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 23:00:19.294767 20113 raft_consensus.cc:729] T 76db18bb9aa3487796175ac7face46a9 P a49447d7b0024e12b538ea065f867cb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a49447d7b0024e12b538ea065f867cb5, State: Initialized, Role: FOLLOWER
I20260807 23:00:19.295387 20113 consensus_queue.cc:260] T 76db18bb9aa3487796175ac7face46a9 P a49447d7b0024e12b538ea065f867cb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "490f1fd977fb4fd586aed08107d9f266" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 36939 } } peers { permanent_uuid: "a49447d7b0024e12b538ea065f867cb5" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 41199 } } peers { permanent_uuid: "8a9f21049c9c4db48d48db896dcce03d" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 45459 } }
I20260807 23:00:19.297055 20083 heartbeater.cc:503] Master 127.25.94.62:40547 was elected leader, sending a full tablet report...
I20260807 23:00:19.296754 20117 raft_consensus.cc:348] T 76db18bb9aa3487796175ac7face46a9 P 490f1fd977fb4fd586aed08107d9f266 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "490f1fd977fb4fd586aed08107d9f266" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 36939 } } peers { permanent_uuid: "a49447d7b0024e12b538ea065f867cb5" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 41199 } } peers { permanent_uuid: "8a9f21049c9c4db48d48db896dcce03d" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 45459 } }
I20260807 23:00:19.297287 20113 ts_tablet_manager.cc:1428] T 76db18bb9aa3487796175ac7face46a9 P a49447d7b0024e12b538ea065f867cb5: Time spent starting tablet: real 0.014s user 0.010s sys 0.006s
I20260807 23:00:19.297284 20117 raft_consensus.cc:374] T 76db18bb9aa3487796175ac7face46a9 P 490f1fd977fb4fd586aed08107d9f266 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 23:00:19.297386 20117 raft_consensus.cc:729] T 76db18bb9aa3487796175ac7face46a9 P 490f1fd977fb4fd586aed08107d9f266 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 490f1fd977fb4fd586aed08107d9f266, State: Initialized, Role: FOLLOWER
I20260807 23:00:19.297993 20117 consensus_queue.cc:260] T 76db18bb9aa3487796175ac7face46a9 P 490f1fd977fb4fd586aed08107d9f266 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "490f1fd977fb4fd586aed08107d9f266" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 36939 } } peers { permanent_uuid: "a49447d7b0024e12b538ea065f867cb5" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 41199 } } peers { permanent_uuid: "8a9f21049c9c4db48d48db896dcce03d" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 45459 } }
I20260807 23:00:19.299198 19868 heartbeater.cc:503] Master 127.25.94.62:40547 was elected leader, sending a full tablet report...
I20260807 23:00:19.300239 20117 ts_tablet_manager.cc:1428] T 76db18bb9aa3487796175ac7face46a9 P 490f1fd977fb4fd586aed08107d9f266: Time spent starting tablet: real 0.016s user 0.004s sys 0.010s
W20260807 23:00:19.328899 19869 tablet.cc:2369] T 76db18bb9aa3487796175ac7face46a9 P 490f1fd977fb4fd586aed08107d9f266: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 23:00:19.379052 20137 raft_consensus.cc:482] T 76db18bb9aa3487796175ac7face46a9 P 490f1fd977fb4fd586aed08107d9f266 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 23:00:19.379513 20137 raft_consensus.cc:504] T 76db18bb9aa3487796175ac7face46a9 P 490f1fd977fb4fd586aed08107d9f266 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "490f1fd977fb4fd586aed08107d9f266" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 36939 } } peers { permanent_uuid: "a49447d7b0024e12b538ea065f867cb5" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 41199 } } peers { permanent_uuid: "8a9f21049c9c4db48d48db896dcce03d" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 45459 } }
I20260807 23:00:19.381454 20137 leader_election.cc:288] T 76db18bb9aa3487796175ac7face46a9 P 490f1fd977fb4fd586aed08107d9f266 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a49447d7b0024e12b538ea065f867cb5 (127.25.94.3:41199), 8a9f21049c9c4db48d48db896dcce03d (127.25.94.1:45459)
I20260807 23:00:19.388828 20111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:00:19.381964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.2:41765 (local address 127.25.94.1:45459)
0807 23:00:19.382288 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0807 23:00:19.382313 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0807 23:00:19.383080 (+ 767us) server_negotiation.cc:407] Connection header received
0807 23:00:19.383284 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:00:19.383288 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:00:19.383349 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:00:19.383449 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 23:00:19.384109 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:00:19.384840 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:00:19.386839 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:00:19.387641 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:00:19.387840 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:00:19.387972 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0807 23:00:19.388386 (+ 414us) server_negotiation.cc:299] Negotiation successful
0807 23:00:19.388520 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156}
I20260807 23:00:19.388995 20140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:00:19.381978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.1:45459 (local address 127.25.94.2:41765)
0807 23:00:19.382968 (+ 990us) negotiation.cc:107] Waiting for socket to connect
0807 23:00:19.382982 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 23:00:19.383132 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 23:00:19.383472 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 23:00:19.383476 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 23:00:19.383512 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 23:00:19.383959 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:00:19.383968 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:00:19.384979 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:00:19.384985 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:00:19.386646 (+ 1661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:00:19.386662 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:00:19.388023 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:00:19.388034 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:00:19.388196 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:00:19.388222 (+ 26us) client_negotiation.cc:769] Sending connection context
0807 23:00:19.388343 (+ 121us) client_negotiation.cc:240] Negotiation successful
0807 23:00:19.388486 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":872,"mutex_wait_us":27,"thread_start_us":651,"threads_started":1}
I20260807 23:00:19.389137 20112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:00:19.381835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.2:39985 (local address 127.25.94.3:41199)
0807 23:00:19.382061 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0807 23:00:19.382070 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 23:00:19.382870 (+ 800us) server_negotiation.cc:407] Connection header received
0807 23:00:19.383113 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:00:19.383117 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:00:19.383178 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:00:19.383278 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 23:00:19.383956 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:00:19.384922 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:00:19.386842 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:00:19.387645 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:00:19.387803 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:00:19.388044 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0807 23:00:19.388460 (+ 416us) server_negotiation.cc:299] Negotiation successful
0807 23:00:19.388853 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260807 23:00:19.389681 20139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:00:19.381860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.3:41199 (local address 127.25.94.2:39985)
0807 23:00:19.382749 (+ 889us) negotiation.cc:107] Waiting for socket to connect
0807 23:00:19.382766 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 23:00:19.382961 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 23:00:19.383294 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 23:00:19.383298 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 23:00:19.383321 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 23:00:19.383783 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:00:19.383802 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:00:19.385057 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:00:19.385064 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:00:19.386670 (+ 1606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:00:19.386683 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:00:19.388034 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:00:19.388045 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:00:19.388251 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:00:19.388279 (+ 28us) client_negotiation.cc:769] Sending connection context
0807 23:00:19.388418 (+ 139us) client_negotiation.cc:240] Negotiation successful
0807 23:00:19.388831 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":755,"mutex_wait_us":29,"spinlock_wait_cycles":34048,"thread_start_us":524,"threads_started":1}
I20260807 23:00:19.390194 19566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76db18bb9aa3487796175ac7face46a9" candidate_uuid: "490f1fd977fb4fd586aed08107d9f266" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9f21049c9c4db48d48db896dcce03d" is_pre_election: true
I20260807 23:00:19.390775 19566 raft_consensus.cc:2454] T 76db18bb9aa3487796175ac7face46a9 P 8a9f21049c9c4db48d48db896dcce03d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 490f1fd977fb4fd586aed08107d9f266 in term 0.
I20260807 23:00:19.390763 20012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76db18bb9aa3487796175ac7face46a9" candidate_uuid: "490f1fd977fb4fd586aed08107d9f266" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a49447d7b0024e12b538ea065f867cb5" is_pre_election: true
I20260807 23:00:19.391244 20012 raft_consensus.cc:2454] T 76db18bb9aa3487796175ac7face46a9 P a49447d7b0024e12b538ea065f867cb5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 490f1fd977fb4fd586aed08107d9f266 in term 0.
I20260807 23:00:19.391569 19668 leader_election.cc:302] T 76db18bb9aa3487796175ac7face46a9 P 490f1fd977fb4fd586aed08107d9f266 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 490f1fd977fb4fd586aed08107d9f266, 8a9f21049c9c4db48d48db896dcce03d; no voters:
I20260807 23:00:19.392071 20137 raft_consensus.cc:2781] T 76db18bb9aa3487796175ac7face46a9 P 490f1fd977fb4fd586aed08107d9f266 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 23:00:19.392228 20137 raft_consensus.cc:482] T 76db18bb9aa3487796175ac7face46a9 P 490f1fd977fb4fd586aed08107d9f266 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 23:00:19.392308 20137 raft_consensus.cc:3037] T 76db18bb9aa3487796175ac7face46a9 P 490f1fd977fb4fd586aed08107d9f266 [term 0 FOLLOWER]: Advancing to term 1
I20260807 23:00:19.394618 20137 raft_consensus.cc:504] T 76db18bb9aa3487796175ac7face46a9 P 490f1fd977fb4fd586aed08107d9f266 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "490f1fd977fb4fd586aed08107d9f266" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 36939 } } peers { permanent_uuid: "a49447d7b0024e12b538ea065f867cb5" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 41199 } } peers { permanent_uuid: "8a9f21049c9c4db48d48db896dcce03d" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 45459 } }
I20260807 23:00:19.395200 20137 leader_election.cc:288] T 76db18bb9aa3487796175ac7face46a9 P 490f1fd977fb4fd586aed08107d9f266 [CANDIDATE]: Term 1 election: Requested vote from peers a49447d7b0024e12b538ea065f867cb5 (127.25.94.3:41199), 8a9f21049c9c4db48d48db896dcce03d (127.25.94.1:45459)
I20260807 23:00:19.395754 19566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76db18bb9aa3487796175ac7face46a9" candidate_uuid: "490f1fd977fb4fd586aed08107d9f266" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9f21049c9c4db48d48db896dcce03d"
I20260807 23:00:19.395772 20012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76db18bb9aa3487796175ac7face46a9" candidate_uuid: "490f1fd977fb4fd586aed08107d9f266" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a49447d7b0024e12b538ea065f867cb5"
I20260807 23:00:19.395977 19566 raft_consensus.cc:3037] T 76db18bb9aa3487796175ac7face46a9 P 8a9f21049c9c4db48d48db896dcce03d [term 0 FOLLOWER]: Advancing to term 1
I20260807 23:00:19.395985 20012 raft_consensus.cc:3037] T 76db18bb9aa3487796175ac7face46a9 P a49447d7b0024e12b538ea065f867cb5 [term 0 FOLLOWER]: Advancing to term 1
W20260807 23:00:19.398749 20086 tablet.cc:2369] T 76db18bb9aa3487796175ac7face46a9 P a49447d7b0024e12b538ea065f867cb5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 23:00:19.398887 20012 raft_consensus.cc:2454] T 76db18bb9aa3487796175ac7face46a9 P a49447d7b0024e12b538ea065f867cb5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 490f1fd977fb4fd586aed08107d9f266 in term 1.
I20260807 23:00:19.399308 19566 raft_consensus.cc:2454] T 76db18bb9aa3487796175ac7face46a9 P 8a9f21049c9c4db48d48db896dcce03d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 490f1fd977fb4fd586aed08107d9f266 in term 1.
I20260807 23:00:19.399374 19667 leader_election.cc:302] T 76db18bb9aa3487796175ac7face46a9 P 490f1fd977fb4fd586aed08107d9f266 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 490f1fd977fb4fd586aed08107d9f266, a49447d7b0024e12b538ea065f867cb5; no voters:
I20260807 23:00:19.399714 20137 raft_consensus.cc:2781] T 76db18bb9aa3487796175ac7face46a9 P 490f1fd977fb4fd586aed08107d9f266 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 23:00:19.400250 20137 raft_consensus.cc:686] T 76db18bb9aa3487796175ac7face46a9 P 490f1fd977fb4fd586aed08107d9f266 [term 1 LEADER]: Becoming Leader. State: Replica: 490f1fd977fb4fd586aed08107d9f266, State: Running, Role: LEADER
I20260807 23:00:19.400732 20137 consensus_queue.cc:237] T 76db18bb9aa3487796175ac7face46a9 P 490f1fd977fb4fd586aed08107d9f266 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "490f1fd977fb4fd586aed08107d9f266" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 36939 } } peers { permanent_uuid: "a49447d7b0024e12b538ea065f867cb5" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 41199 } } peers { permanent_uuid: "8a9f21049c9c4db48d48db896dcce03d" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 45459 } }
I20260807 23:00:19.405730 19314 catalog_manager.cc:5495] T 76db18bb9aa3487796175ac7face46a9 P 490f1fd977fb4fd586aed08107d9f266 reported cstate change: term changed from 0 to 1, leader changed from <none> to 490f1fd977fb4fd586aed08107d9f266 (127.25.94.2). New cstate: current_term: 1 leader_uuid: "490f1fd977fb4fd586aed08107d9f266" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "490f1fd977fb4fd586aed08107d9f266" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 36939 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a49447d7b0024e12b538ea065f867cb5" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 41199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a9f21049c9c4db48d48db896dcce03d" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 45459 } health_report { overall_health: UNKNOWN } } }
I20260807 23:00:19.449368 20109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:00:19.443757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48650 (local address 127.25.94.2:36939)
0807 23:00:19.443999 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0807 23:00:19.444008 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 23:00:19.444028 (+ 20us) server_negotiation.cc:407] Connection header received
0807 23:00:19.444132 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:00:19.444136 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:00:19.444227 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:00:19.444399 (+ 172us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 23:00:19.444808 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:00:19.445579 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:00:19.447054 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:00:19.447479 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:00:19.447627 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:00:19.448141 (+ 514us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 23:00:19.448686 (+ 545us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 23:00:19.448852 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0807 23:00:19.448932 (+ 80us) server_negotiation.cc:299] Negotiation successful
0807 23:00:19.449019 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260807 23:00:19.457887 19566 raft_consensus.cc:1261] T 76db18bb9aa3487796175ac7face46a9 P 8a9f21049c9c4db48d48db896dcce03d [term 1 FOLLOWER]: Refusing update from remote peer 490f1fd977fb4fd586aed08107d9f266: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 23:00:19.458004 20012 raft_consensus.cc:1261] T 76db18bb9aa3487796175ac7face46a9 P a49447d7b0024e12b538ea065f867cb5 [term 1 FOLLOWER]: Refusing update from remote peer 490f1fd977fb4fd586aed08107d9f266: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 23:00:19.459013 20141 consensus_queue.cc:1035] T 76db18bb9aa3487796175ac7face46a9 P 490f1fd977fb4fd586aed08107d9f266 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a49447d7b0024e12b538ea065f867cb5" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 41199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 23:00:19.459367 20137 consensus_queue.cc:1035] T 76db18bb9aa3487796175ac7face46a9 P 490f1fd977fb4fd586aed08107d9f266 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a9f21049c9c4db48d48db896dcce03d" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 45459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 23:00:19.474833 20152 mvcc.cc:205] Tried to move back new op lower bound from 7316044265288298496 to 7316044265069330432. Current Snapshot: MvccSnapshot[applied={T|T < 7316044265288298496}]
I20260807 23:00:19.486768 19314 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40416:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 23:00:19.487053 19314 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40416:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 23:00:19.490747 19314 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7cf6aa251b54475d819168a882c99955: Sending DeleteTablet for 3 replicas of tablet 76db18bb9aa3487796175ac7face46a9
I20260807 23:00:19.491957 19979 tablet_service.cc:1514] Processing DeleteTablet for tablet 76db18bb9aa3487796175ac7face46a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 23:00:19 UTC) from {username='kudu', principal='kudu/127.25.94.62@KRBTEST.COM'} at 127.0.0.1:32950
I20260807 23:00:19.491951 19755 tablet_service.cc:1514] Processing DeleteTablet for tablet 76db18bb9aa3487796175ac7face46a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 23:00:19 UTC) from {username='kudu', principal='kudu/127.25.94.62@KRBTEST.COM'} at 127.0.0.1:48634
I20260807 23:00:19.491984 19526 tablet_service.cc:1514] Processing DeleteTablet for tablet 76db18bb9aa3487796175ac7face46a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 23:00:19 UTC) from {username='kudu', principal='kudu/127.25.94.62@KRBTEST.COM'} at 127.0.0.1:60160
I20260807 23:00:19.492967 20170 tablet_replica.cc:330] T 76db18bb9aa3487796175ac7face46a9 P 490f1fd977fb4fd586aed08107d9f266: stopping tablet replica
I20260807 23:00:19.493340 20167 tablet_replica.cc:330] T 76db18bb9aa3487796175ac7face46a9 P a49447d7b0024e12b538ea065f867cb5: stopping tablet replica
I20260807 23:00:19.493465 20170 raft_consensus.cc:2229] T 76db18bb9aa3487796175ac7face46a9 P 490f1fd977fb4fd586aed08107d9f266 [term 1 LEADER]: Raft consensus shutting down.
I20260807 23:00:19.493876 20167 raft_consensus.cc:2229] T 76db18bb9aa3487796175ac7face46a9 P a49447d7b0024e12b538ea065f867cb5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 23:00:19.494176 20170 raft_consensus.cc:2258] T 76db18bb9aa3487796175ac7face46a9 P 490f1fd977fb4fd586aed08107d9f266 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 23:00:19.494256 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 19418
I20260807 23:00:19.494561 20167 raft_consensus.cc:2258] T 76db18bb9aa3487796175ac7face46a9 P a49447d7b0024e12b538ea065f867cb5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 23:00:19.494944 20169 tablet_replica.cc:330] T 76db18bb9aa3487796175ac7face46a9 P 8a9f21049c9c4db48d48db896dcce03d: stopping tablet replica
I20260807 23:00:19.495361 20169 raft_consensus.cc:2229] T 76db18bb9aa3487796175ac7face46a9 P 8a9f21049c9c4db48d48db896dcce03d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 23:00:19.495888 20169 raft_consensus.cc:2258] T 76db18bb9aa3487796175ac7face46a9 P 8a9f21049c9c4db48d48db896dcce03d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 23:00:19.497622 20167 ts_tablet_manager.cc:1905] T 76db18bb9aa3487796175ac7face46a9 P a49447d7b0024e12b538ea065f867cb5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 23:00:19.498075 20170 ts_tablet_manager.cc:1905] T 76db18bb9aa3487796175ac7face46a9 P 490f1fd977fb4fd586aed08107d9f266: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 23:00:19.506354 20170 ts_tablet_manager.cc:1918] T 76db18bb9aa3487796175ac7face46a9 P 490f1fd977fb4fd586aed08107d9f266: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 23:00:19.506554 20170 log.cc:1196] T 76db18bb9aa3487796175ac7face46a9 P 490f1fd977fb4fd586aed08107d9f266: Deleting WAL directory at /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-1/wal/wals/76db18bb9aa3487796175ac7face46a9
I20260807 23:00:19.506914 20167 ts_tablet_manager.cc:1918] T 76db18bb9aa3487796175ac7face46a9 P a49447d7b0024e12b538ea065f867cb5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 23:00:19.507159 20170 ts_tablet_manager.cc:1939] T 76db18bb9aa3487796175ac7face46a9 P 490f1fd977fb4fd586aed08107d9f266: Deleting consensus metadata
I20260807 23:00:19.507187 20167 log.cc:1196] T 76db18bb9aa3487796175ac7face46a9 P a49447d7b0024e12b538ea065f867cb5: Deleting WAL directory at /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-2/wal/wals/76db18bb9aa3487796175ac7face46a9
I20260807 23:00:19.507845 20167 ts_tablet_manager.cc:1939] T 76db18bb9aa3487796175ac7face46a9 P a49447d7b0024e12b538ea065f867cb5: Deleting consensus metadata
I20260807 23:00:19.509132 19293 catalog_manager.cc:4841] TS 490f1fd977fb4fd586aed08107d9f266 (127.25.94.2:36939): tablet 76db18bb9aa3487796175ac7face46a9 (table test-table [id=7284e103ec9a4959ad52fd136ba85b91]) successfully deleted
I20260807 23:00:19.509833 19284 catalog_manager.cc:4841] TS a49447d7b0024e12b538ea065f867cb5 (127.25.94.3:41199): tablet 76db18bb9aa3487796175ac7face46a9 (table test-table [id=7284e103ec9a4959ad52fd136ba85b91]) successfully deleted
I20260807 23:00:19.510686 20169 ts_tablet_manager.cc:1905] T 76db18bb9aa3487796175ac7face46a9 P 8a9f21049c9c4db48d48db896dcce03d: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 07 23:00:19 dist-test-slave-z8x9 krb5kdc[19243](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143616, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.1@KRBTEST.COM
I20260807 23:00:19.518092 20169 ts_tablet_manager.cc:1918] T 76db18bb9aa3487796175ac7face46a9 P 8a9f21049c9c4db48d48db896dcce03d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 23:00:19.518292 20169 log.cc:1196] T 76db18bb9aa3487796175ac7face46a9 P 8a9f21049c9c4db48d48db896dcce03d: Deleting WAL directory at /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-0/wal/wals/76db18bb9aa3487796175ac7face46a9
I20260807 23:00:19.518922 20169 ts_tablet_manager.cc:1939] T 76db18bb9aa3487796175ac7face46a9 P 8a9f21049c9c4db48d48db896dcce03d: Deleting consensus metadata
I20260807 23:00:19.520972 19288 catalog_manager.cc:4841] TS 8a9f21049c9c4db48d48db896dcce03d (127.25.94.1:45459): tablet 76db18bb9aa3487796175ac7face46a9 (table test-table [id=7284e103ec9a4959ad52fd136ba85b91]) successfully deleted
I20260807 23:00:19.523433 20111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:00:19.495470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60170 (local address 127.25.94.1:45459)
0807 23:00:19.495671 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0807 23:00:19.495684 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0807 23:00:19.506038 (+ 10354us) server_negotiation.cc:407] Connection header received
0807 23:00:19.506545 (+ 507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:00:19.506551 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:00:19.506654 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:00:19.506739 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:00:19.509033 (+ 2294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:00:19.510262 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:00:19.511907 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:00:19.512348 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:00:19.512506 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:00:19.516884 (+ 4378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:00:19.516897 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:00:19.516912 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 23:00:19.516947 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:00:19.520250 (+ 3303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:00:19.521286 (+ 1036us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:00:19.521301 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:00:19.521304 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:00:19.521438 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:00:19.521825 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:00:19.521832 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:00:19.521835 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:00:19.522196 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:00:19.522527 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0807 23:00:19.522861 (+ 334us) server_negotiation.cc:299] Negotiation successful
0807 23:00:19.523074 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260807 23:00:19.524395 19622 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 23:00:19.633332 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 19418
I20260807 23:00:19.658773 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 19641
Aug 07 23:00:19 dist-test-slave-z8x9 krb5kdc[19243](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143616, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.2@KRBTEST.COM
I20260807 23:00:19.676445 20109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:00:19.659787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48662 (local address 127.25.94.2:36939)
0807 23:00:19.660010 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0807 23:00:19.660023 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0807 23:00:19.660042 (+ 19us) server_negotiation.cc:407] Connection header received
0807 23:00:19.660108 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:00:19.660111 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:00:19.660171 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:00:19.660304 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:00:19.661781 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:00:19.663107 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:00:19.664522 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:00:19.664954 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:00:19.665103 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:00:19.670032 (+ 4929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:00:19.670047 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:00:19.670063 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 23:00:19.670116 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:00:19.673400 (+ 3284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:00:19.674134 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:00:19.674151 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:00:19.674153 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:00:19.674314 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:00:19.674721 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:00:19.674726 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:00:19.674732 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:00:19.675140 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:00:19.675492 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0807 23:00:19.675811 (+ 319us) server_negotiation.cc:299] Negotiation successful
0807 23:00:19.676061 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260807 23:00:19.677322 19860 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 23:00:19.796121 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 19641
I20260807 23:00:19.828190 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 19876
Aug 07 23:00:19 dist-test-slave-z8x9 krb5kdc[19243](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143616, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.3@KRBTEST.COM
I20260807 23:00:19.848810 20112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:00:19.829043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32962 (local address 127.25.94.3:41199)
0807 23:00:19.829287 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0807 23:00:19.829292 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 23:00:19.829323 (+ 31us) server_negotiation.cc:407] Connection header received
0807 23:00:19.829466 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:00:19.829471 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:00:19.829536 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:00:19.829621 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:00:19.831133 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:00:19.831896 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:00:19.833528 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:00:19.833929 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:00:19.834030 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:00:19.842331 (+ 8301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:00:19.842351 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:00:19.842369 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 23:00:19.842421 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:00:19.845682 (+ 3261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:00:19.846521 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:00:19.846529 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:00:19.846532 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:00:19.846682 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:00:19.847120 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:00:19.847124 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:00:19.847126 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:00:19.847496 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:00:19.847866 (+ 370us) server_negotiation.cc:1035] Waiting for connection context
0807 23:00:19.848208 (+ 342us) server_negotiation.cc:299] Negotiation successful
0807 23:00:19.848433 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260807 23:00:19.849570 20075 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 23:00:19.939620 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 19876
I20260807 23:00:19.963778 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 19261
I20260807 23:00:19.964999 19371 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 23:00:20.076848 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 19261
I20260807 23:00:40.106359 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.62:40547
--webserver_interface=127.25.94.62
--webserver_port=44809
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.62.keytab
--principal=kudu/127.25.94.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.20:35499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.94.62:40547
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 23:00:40.500182 20179 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 23:00:40.500559 20179 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 23:00:40.500720 20179 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 23:00:40.508258 20179 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260807 23:00:40.508420 20179 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260807 23:00:40.508507 20179 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260807 23:00:40.508599 20179 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 23:00:40.508699 20179 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 23:00:40.508816 20179 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 23:00:40.508920 20179 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 23:00:40.509006 20179 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 23:00:40.519172 20179 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:35499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.25.94.62:40547
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.62.keytab
--principal=kudu/127.25.94.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.62:40547
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.94.62
--webserver_port=44809
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 23:00:40.520581 20179 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 23:00:40.522007 20179 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 23:00:40.548997 20189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 23:00:40.549273 20197 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 23:00:40.549737 20194 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 23:00:40.550449 20179 server_base.cc:1034] running on GCE node
I20260807 23:00:40.551553 20179 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 23:00:40.553417 20179 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 23:00:40.554675 20179 hybrid_clock.cc:648] HybridClock initialized: now 1786143640554611 us; error 49 us; skew 500 ppm
I20260807 23:00:40.555094 20179 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 23:00:40 dist-test-slave-z8x9 krb5kdc[19243](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143640, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 23:00:40.559087 20179 init.cc:377] Logged in from keytab as kudu/127.25.94.62@KRBTEST.COM (short username kudu)
I20260807 23:00:40.560948 20179 webserver.cc:460] Webserver started at http://127.25.94.62:44809/ using document root <none> and password file <none>
I20260807 23:00:40.561839 20179 fs_manager.cc:359] Metadata directory not provided
I20260807 23:00:40.562000 20179 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 23:00:40.568291 20179 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.001s
I20260807 23:00:40.571580 20206 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 23:00:40.573468 20179 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 23:00:40.573679 20179 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/master-0/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/master-0/wal
uuid: "7cf6aa251b54475d819168a882c99955"
format_stamp: "Formatted at 2026-08-07 23:00:17 on dist-test-slave-z8x9"
server_key: "bc426fdb20e3d3a60185290f897b98e4"
server_key_iv: "388cc39b6271fbc10782fbce5d3e08ff"
server_key_version: "encryptionkey@0"
I20260807 23:00:40.574823 20179 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 23:00:40.598121 20179 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 23:00:40.602790 20179 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 23:00:40.603114 20179 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 23:00:40.623724 20179 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.62:40547
I20260807 23:00:40.623756 20293 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.62:40547 every 8 connection(s)
I20260807 23:00:40.625479 20179 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
I20260807 23:00:40.632964 20294 sys_catalog.cc:263] Verifying existing consensus state
I20260807 23:00:40.633147 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 20179
I20260807 23:00:40.634318 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.1:45459
--local_ip_for_outbound_sockets=127.25.94.1
--tserver_master_addrs=127.25.94.62:40547
--webserver_port=35209
--webserver_interface=127.25.94.1
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.1.keytab
--principal=kudu/127.25.94.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.20:35499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 23:00:40.638257 20294 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7cf6aa251b54475d819168a882c99955: Bootstrap starting.
I20260807 23:00:40.662132 20294 log.cc:824] T 00000000000000000000000000000000 P 7cf6aa251b54475d819168a882c99955: Log is configured to *not* fsync() on all Append() calls
I20260807 23:00:40.677052 20294 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7cf6aa251b54475d819168a882c99955: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 23:00:40.677690 20294 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7cf6aa251b54475d819168a882c99955: Bootstrap complete.
I20260807 23:00:40.687618 20294 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7cf6aa251b54475d819168a882c99955 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cf6aa251b54475d819168a882c99955" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 40547 } }
I20260807 23:00:40.688508 20294 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7cf6aa251b54475d819168a882c99955 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7cf6aa251b54475d819168a882c99955, State: Initialized, Role: FOLLOWER
I20260807 23:00:40.689335 20294 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7cf6aa251b54475d819168a882c99955 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cf6aa251b54475d819168a882c99955" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 40547 } }
I20260807 23:00:40.689589 20294 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7cf6aa251b54475d819168a882c99955 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 23:00:40.689714 20294 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7cf6aa251b54475d819168a882c99955 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 23:00:40.689920 20294 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7cf6aa251b54475d819168a882c99955 [term 1 FOLLOWER]: Advancing to term 2
I20260807 23:00:40.693902 20294 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7cf6aa251b54475d819168a882c99955 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cf6aa251b54475d819168a882c99955" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 40547 } }
I20260807 23:00:40.694473 20294 leader_election.cc:302] T 00000000000000000000000000000000 P 7cf6aa251b54475d819168a882c99955 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7cf6aa251b54475d819168a882c99955; no voters:
I20260807 23:00:40.695176 20294 leader_election.cc:288] T 00000000000000000000000000000000 P 7cf6aa251b54475d819168a882c99955 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260807 23:00:40.695325 20303 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7cf6aa251b54475d819168a882c99955 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 23:00:40.696596 20303 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7cf6aa251b54475d819168a882c99955 [term 2 LEADER]: Becoming Leader. State: Replica: 7cf6aa251b54475d819168a882c99955, State: Running, Role: LEADER
I20260807 23:00:40.697225 20303 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7cf6aa251b54475d819168a882c99955 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cf6aa251b54475d819168a882c99955" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 40547 } }
I20260807 23:00:40.697675 20294 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7cf6aa251b54475d819168a882c99955 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 23:00:40.699916 20304 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7cf6aa251b54475d819168a882c99955 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "7cf6aa251b54475d819168a882c99955" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cf6aa251b54475d819168a882c99955" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 40547 } } }
I20260807 23:00:40.700408 20304 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7cf6aa251b54475d819168a882c99955 [sys.catalog]: This master's current role is: LEADER
I20260807 23:00:40.701221 20308 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7cf6aa251b54475d819168a882c99955 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7cf6aa251b54475d819168a882c99955. Latest consensus state: current_term: 2 leader_uuid: "7cf6aa251b54475d819168a882c99955" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cf6aa251b54475d819168a882c99955" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 40547 } } }
I20260807 23:00:40.701670 20308 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7cf6aa251b54475d819168a882c99955 [sys.catalog]: This master's current role is: LEADER
I20260807 23:00:40.704542 20311 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 23:00:40.713400 20311 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 23:00:40.714886 20311 catalog_manager.cc:1259] Loaded cluster ID: 7a21efc46e6c4d0689df4ef042c69b80
I20260807 23:00:40.715073 20311 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 23:00:40.719250 20311 catalog_manager.cc:1504] Loading token signing keys...
I20260807 23:00:40.721123 20311 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 7cf6aa251b54475d819168a882c99955: Loaded TSK: 0
I20260807 23:00:40.733031 20329 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260807 23:00:40.753682 20311 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7cf6aa251b54475d819168a882c99955: Generated new TSK 1
I20260807 23:00:40.754128 20311 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260807 23:00:41.008822 20296 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 23:00:41.009196 20296 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 23:00:41.009366 20296 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 23:00:41.020084 20296 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 23:00:41.020256 20296 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 23:00:41.020542 20296 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.1
I20260807 23:00:41.032610 20296 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:35499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.1.keytab
--principal=kudu/127.25.94.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.1:45459
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.94.1
--webserver_port=35209
--tserver_master_addrs=127.25.94.62:40547
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.1
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 23:00:41.034041 20296 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 23:00:41.035378 20296 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 23:00:41.050462 20343 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 23:00:41.050540 20338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 23:00:41.050515 20346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 23:00:41.052551 20296 server_base.cc:1034] running on GCE node
I20260807 23:00:41.053323 20296 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 23:00:41.054358 20296 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 23:00:41.055620 20296 hybrid_clock.cc:648] HybridClock initialized: now 1786143641055555 us; error 51 us; skew 500 ppm
I20260807 23:00:41.055985 20296 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 23:00:41 dist-test-slave-z8x9 krb5kdc[19243](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143641, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 23:00:41.059554 20296 init.cc:377] Logged in from keytab as kudu/127.25.94.1@KRBTEST.COM (short username kudu)
I20260807 23:00:41.061480 20296 webserver.cc:460] Webserver started at http://127.25.94.1:35209/ using document root <none> and password file <none>
I20260807 23:00:41.062285 20296 fs_manager.cc:359] Metadata directory not provided
I20260807 23:00:41.062461 20296 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 23:00:41.068626 20296 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.002s
I20260807 23:00:41.071879 20362 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 23:00:41.073527 20296 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 23:00:41.073680 20296 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-0/wal
uuid: "8a9f21049c9c4db48d48db896dcce03d"
format_stamp: "Formatted at 2026-08-07 23:00:17 on dist-test-slave-z8x9"
server_key: "fe42c621df0923528e732238bb21d9b0"
server_key_iv: "3420d0b1ab3d2891a30b640606475760"
server_key_version: "encryptionkey@0"
I20260807 23:00:41.074698 20296 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 23:00:41.102705 20296 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 23:00:41.108013 20296 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 23:00:41.108273 20296 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 23:00:41.109380 20296 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 23:00:41.110730 20296 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 23:00:41.110822 20296 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 23:00:41.110908 20296 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 23:00:41.110963 20296 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 23:00:41.147850 20296 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.1:45459
I20260807 23:00:41.148227 20544 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.1:45459 every 8 connection(s)
I20260807 23:00:41.149631 20296 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-0/data/info.pb
I20260807 23:00:41.160204 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 20296
I20260807 23:00:41.161103 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.2:36939
--local_ip_for_outbound_sockets=127.25.94.2
--tserver_master_addrs=127.25.94.62:40547
--webserver_port=37971
--webserver_interface=127.25.94.2
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.2.keytab
--principal=kudu/127.25.94.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.20:35499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 07 23:00:41 dist-test-slave-z8x9 krb5kdc[19243](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143641, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.1@KRBTEST.COM for kudu/127.25.94.62@KRBTEST.COM
I20260807 23:00:41.181842 20551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:00:41.153155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:40547 (local address 127.25.94.1:53525)
0807 23:00:41.154249 (+ 1094us) negotiation.cc:107] Waiting for socket to connect
0807 23:00:41.154299 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0807 23:00:41.156168 (+ 1869us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 23:00:41.157993 (+ 1825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 23:00:41.158002 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 23:00:41.158595 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0807 23:00:41.159225 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:00:41.159247 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:00:41.162126 (+ 2879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:00:41.162131 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:00:41.163951 (+ 1820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:00:41.163973 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:00:41.165680 (+ 1707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:00:41.165690 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:00:41.166028 (+ 338us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:00:41.166900 (+ 872us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 23:00:41.166935 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0807 23:00:41.170140 (+ 3205us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 23:00:41.174928 (+ 4788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 23:00:41.174941 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 23:00:41.174962 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0807 23:00:41.175749 (+ 787us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 23:00:41.176922 (+ 1173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 23:00:41.176931 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 23:00:41.176937 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 23:00:41.177369 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 23:00:41.178515 (+ 1146us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 23:00:41.178524 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 23:00:41.179006 (+ 482us) client_negotiation.cc:769] Sending connection context
0807 23:00:41.179577 (+ 571us) client_negotiation.cc:240] Negotiation successful
0807 23:00:41.180287 (+ 710us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":715,"thread_start_us":507,"threads_started":1}
I20260807 23:00:41.181842 20549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:00:41.152998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.1:53525 (local address 127.25.94.62:40547)
0807 23:00:41.153737 (+ 739us) server_negotiation.cc:206] Beginning negotiation
0807 23:00:41.153750 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0807 23:00:41.154463 (+ 713us) server_negotiation.cc:407] Connection header received
0807 23:00:41.156807 (+ 2344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:00:41.156822 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:00:41.157327 (+ 505us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:00:41.157669 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:00:41.159558 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:00:41.161844 (+ 2286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:00:41.164165 (+ 2321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:00:41.165419 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:00:41.165652 (+ 233us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:00:41.170590 (+ 4938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:00:41.170604 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:00:41.170628 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 23:00:41.170686 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:00:41.174681 (+ 3995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:00:41.176220 (+ 1539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:00:41.176234 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:00:41.176245 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:00:41.176668 (+ 423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:00:41.177587 (+ 919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:00:41.177595 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:00:41.177597 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:00:41.178262 (+ 665us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:00:41.178576 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0807 23:00:41.179968 (+ 1392us) server_negotiation.cc:299] Negotiation successful
0807 23:00:41.180543 (+ 575us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":532,"thread_start_us":387,"threads_started":1}
I20260807 23:00:41.185343 20546 heartbeater.cc:347] Connected to a master server at 127.25.94.62:40547
I20260807 23:00:41.185738 20546 heartbeater.cc:464] Registering TS with master...
I20260807 23:00:41.186825 20546 heartbeater.cc:511] Master 127.25.94.62:40547 requested a full tablet report, sending...
I20260807 23:00:41.190630 20238 ts_manager.cc:194] Registered new tserver with Master: 8a9f21049c9c4db48d48db896dcce03d (127.25.94.1:45459)
I20260807 23:00:41.192448 20238 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.94.1@KRBTEST.COM'} at 127.25.94.1:53525
W20260807 23:00:41.518500 20554 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 23:00:41.518791 20554 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 23:00:41.518947 20554 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 23:00:41.526324 20554 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 23:00:41.526432 20554 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 23:00:41.526624 20554 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.2
I20260807 23:00:41.536495 20554 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:35499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.2.keytab
--principal=kudu/127.25.94.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.2:36939
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.94.2
--webserver_port=37971
--tserver_master_addrs=127.25.94.62:40547
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.2
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 23:00:41.537842 20554 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 23:00:41.539170 20554 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 23:00:41.553159 20562 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 23:00:41.553059 20564 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 23:00:41.553097 20566 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 23:00:41.554352 20554 server_base.cc:1034] running on GCE node
I20260807 23:00:41.555059 20554 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 23:00:41.556097 20554 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 23:00:41.557292 20554 hybrid_clock.cc:648] HybridClock initialized: now 1786143641557231 us; error 56 us; skew 500 ppm
I20260807 23:00:41.557673 20554 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 23:00:41 dist-test-slave-z8x9 krb5kdc[19243](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143641, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 23:00:41.560760 20554 init.cc:377] Logged in from keytab as kudu/127.25.94.2@KRBTEST.COM (short username kudu)
I20260807 23:00:41.562173 20554 webserver.cc:460] Webserver started at http://127.25.94.2:37971/ using document root <none> and password file <none>
I20260807 23:00:41.562845 20554 fs_manager.cc:359] Metadata directory not provided
I20260807 23:00:41.562965 20554 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 23:00:41.568298 20554 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260807 23:00:41.570988 20577 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 23:00:41.572346 20554 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260807 23:00:41.572523 20554 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-1/wal
uuid: "490f1fd977fb4fd586aed08107d9f266"
format_stamp: "Formatted at 2026-08-07 23:00:18 on dist-test-slave-z8x9"
server_key: "ad85a15d8b332a3dd74c4377ce274f8a"
server_key_iv: "6ea30c2da1c71d3dd026ef2728b7c3bb"
server_key_version: "encryptionkey@0"
I20260807 23:00:41.573587 20554 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 23:00:41.594576 20554 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 23:00:41.598958 20554 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 23:00:41.599267 20554 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 23:00:41.600304 20554 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 23:00:41.601764 20554 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 23:00:41.601873 20554 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 23:00:41.602008 20554 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 23:00:41.602097 20554 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 23:00:41.637275 20554 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.2:36939
I20260807 23:00:41.637331 20777 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.2:36939 every 8 connection(s)
I20260807 23:00:41.638710 20554 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-1/data/info.pb
I20260807 23:00:41.647312 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 20554
I20260807 23:00:41.648020 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.3:41199
--local_ip_for_outbound_sockets=127.25.94.3
--tserver_master_addrs=127.25.94.62:40547
--webserver_port=33175
--webserver_interface=127.25.94.3
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.3.keytab
--principal=kudu/127.25.94.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.20:35499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 07 23:00:41 dist-test-slave-z8x9 krb5kdc[19243](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143641, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.2@KRBTEST.COM for kudu/127.25.94.62@KRBTEST.COM
I20260807 23:00:41.661640 20549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:00:41.641665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.2:40569 (local address 127.25.94.62:40547)
0807 23:00:41.641860 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0807 23:00:41.641865 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 23:00:41.643049 (+ 1184us) server_negotiation.cc:407] Connection header received
0807 23:00:41.645125 (+ 2076us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:00:41.645131 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:00:41.645315 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:00:41.645406 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:00:41.646915 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:00:41.647764 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:00:41.650336 (+ 2572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:00:41.650761 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:00:41.650859 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:00:41.655378 (+ 4519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:00:41.655383 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:00:41.655387 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 23:00:41.655419 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:00:41.658166 (+ 2747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:00:41.659108 (+ 942us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:00:41.659112 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:00:41.659114 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:00:41.659233 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:00:41.659813 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:00:41.659818 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:00:41.659822 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:00:41.660146 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:00:41.660318 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0807 23:00:41.660996 (+ 678us) server_negotiation.cc:299] Negotiation successful
0807 23:00:41.661265 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260807 23:00:41.662674 20786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:00:41.642126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:40547 (local address 127.25.94.2:40569)
0807 23:00:41.642879 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0807 23:00:41.642924 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0807 23:00:41.644856 (+ 1932us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 23:00:41.645633 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 23:00:41.645641 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 23:00:41.646168 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0807 23:00:41.646742 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:00:41.646759 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:00:41.647930 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:00:41.647936 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:00:41.650194 (+ 2258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:00:41.650211 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:00:41.650999 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:00:41.651010 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:00:41.651345 (+ 335us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:00:41.652092 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 23:00:41.652132 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0807 23:00:41.655136 (+ 3004us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 23:00:41.658306 (+ 3170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 23:00:41.658318 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 23:00:41.658339 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0807 23:00:41.658993 (+ 654us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 23:00:41.659357 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 23:00:41.659364 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 23:00:41.659371 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0807 23:00:41.659658 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 23:00:41.660281 (+ 623us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 23:00:41.660289 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 23:00:41.660641 (+ 352us) client_negotiation.cc:769] Sending connection context
0807 23:00:41.660921 (+ 280us) client_negotiation.cc:240] Negotiation successful
0807 23:00:41.661474 (+ 553us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":309,"threads_started":1}
I20260807 23:00:41.664225 20781 heartbeater.cc:347] Connected to a master server at 127.25.94.62:40547
I20260807 23:00:41.664533 20781 heartbeater.cc:464] Registering TS with master...
I20260807 23:00:41.665297 20781 heartbeater.cc:511] Master 127.25.94.62:40547 requested a full tablet report, sending...
I20260807 23:00:41.666769 20238 ts_manager.cc:194] Registered new tserver with Master: 490f1fd977fb4fd586aed08107d9f266 (127.25.94.2:36939)
I20260807 23:00:41.667727 20238 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.94.2@KRBTEST.COM'} at 127.25.94.2:40569
W20260807 23:00:42.005014 20787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 23:00:42.005416 20787 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 23:00:42.005611 20787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 23:00:42.013643 20787 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 23:00:42.013806 20787 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 23:00:42.014014 20787 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.3
I20260807 23:00:42.024494 20787 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:35499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.3.keytab
--principal=kudu/127.25.94.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.3:41199
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.94.3
--webserver_port=33175
--tserver_master_addrs=127.25.94.62:40547
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.3
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 23:00:42.025956 20787 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 23:00:42.027302 20787 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 23:00:42.041091 20793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 23:00:42.041159 20794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 23:00:42.041091 20797 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 23:00:42.042443 20787 server_base.cc:1034] running on GCE node
I20260807 23:00:42.043164 20787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 23:00:42.044225 20787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 23:00:42.045463 20787 hybrid_clock.cc:648] HybridClock initialized: now 1786143642045394 us; error 67 us; skew 500 ppm
I20260807 23:00:42.045802 20787 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 23:00:42 dist-test-slave-z8x9 krb5kdc[19243](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143642, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 23:00:42.049108 20787 init.cc:377] Logged in from keytab as kudu/127.25.94.3@KRBTEST.COM (short username kudu)
I20260807 23:00:42.050711 20787 webserver.cc:460] Webserver started at http://127.25.94.3:33175/ using document root <none> and password file <none>
I20260807 23:00:42.051430 20787 fs_manager.cc:359] Metadata directory not provided
I20260807 23:00:42.051602 20787 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 23:00:42.057080 20787 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260807 23:00:42.060166 20809 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 23:00:42.061594 20787 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260807 23:00:42.061779 20787 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-2/wal
uuid: "a49447d7b0024e12b538ea065f867cb5"
format_stamp: "Formatted at 2026-08-07 23:00:19 on dist-test-slave-z8x9"
server_key: "104d48c30a3e317d64e2257306c7c74f"
server_key_iv: "a256b9d5ac16994f5bb2160b184176ec"
server_key_version: "encryptionkey@0"
I20260807 23:00:42.062856 20787 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 23:00:42.081842 20787 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 23:00:42.086612 20787 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 23:00:42.086925 20787 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 23:00:42.088019 20787 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 23:00:42.089625 20787 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 23:00:42.089759 20787 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 23:00:42.089921 20787 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 23:00:42.090031 20787 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 23:00:42.130824 20787 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.3:41199
I20260807 23:00:42.130945 21025 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.3:41199 every 8 connection(s)
I20260807 23:00:42.132514 20787 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-2/data/info.pb
I20260807 23:00:42.141716 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 20787
Aug 07 23:00:42 dist-test-slave-z8x9 krb5kdc[19243](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143642, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.3@KRBTEST.COM for kudu/127.25.94.62@KRBTEST.COM
I20260807 23:00:42.157948 20549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:00:42.135588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.3:51535 (local address 127.25.94.62:40547)
0807 23:00:42.135874 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0807 23:00:42.135881 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 23:00:42.137103 (+ 1222us) server_negotiation.cc:407] Connection header received
0807 23:00:42.139126 (+ 2023us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:00:42.139136 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:00:42.139242 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:00:42.139376 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:00:42.141440 (+ 2064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:00:42.142538 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:00:42.144890 (+ 2352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:00:42.145480 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:00:42.145632 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:00:42.150545 (+ 4913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:00:42.150554 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:00:42.150563 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 23:00:42.150605 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:00:42.154250 (+ 3645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:00:42.155059 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:00:42.155069 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:00:42.155071 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:00:42.155197 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:00:42.155686 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:00:42.155690 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:00:42.155692 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:00:42.156023 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:00:42.156198 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0807 23:00:42.157297 (+ 1099us) server_negotiation.cc:299] Negotiation successful
0807 23:00:42.157561 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"spinlock_wait_cycles":12544}
I20260807 23:00:42.159132 21030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:00:42.136043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:40547 (local address 127.25.94.3:51535)
0807 23:00:42.136919 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0807 23:00:42.136959 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0807 23:00:42.138819 (+ 1860us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 23:00:42.139667 (+ 848us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 23:00:42.139676 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 23:00:42.140497 (+ 821us) client_negotiation.cc:189] Negotiated authn=SASL
0807 23:00:42.141161 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:00:42.141248 (+ 87us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:00:42.143401 (+ 2153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:00:42.143405 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:00:42.144718 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:00:42.144733 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:00:42.145747 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:00:42.145756 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:00:42.146034 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:00:42.146781 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 23:00:42.146841 (+ 60us) client_negotiation.cc:656] calling sasl_client_start()
0807 23:00:42.150201 (+ 3360us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 23:00:42.154415 (+ 4214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 23:00:42.154425 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 23:00:42.154442 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0807 23:00:42.154915 (+ 473us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 23:00:42.155332 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 23:00:42.155341 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 23:00:42.155345 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 23:00:42.155552 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 23:00:42.156171 (+ 619us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 23:00:42.156179 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 23:00:42.156670 (+ 491us) client_negotiation.cc:769] Sending connection context
0807 23:00:42.157015 (+ 345us) client_negotiation.cc:240] Negotiation successful
0807 23:00:42.157661 (+ 646us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":380,"threads_started":1}
I20260807 23:00:42.159353 21032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:00:42.142919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48828 (local address 127.25.94.62:40547)
0807 23:00:42.143469 (+ 550us) server_negotiation.cc:206] Beginning negotiation
0807 23:00:42.143475 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 23:00:42.143587 (+ 112us) server_negotiation.cc:407] Connection header received
0807 23:00:42.143786 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:00:42.143792 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:00:42.143876 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:00:42.143987 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:00:42.145981 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:00:42.146730 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:00:42.148505 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:00:42.149256 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:00:42.149390 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:00:42.152816 (+ 3426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:00:42.152824 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:00:42.152830 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 23:00:42.152886 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:00:42.156178 (+ 3292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:00:42.157076 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:00:42.157082 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:00:42.157085 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:00:42.157304 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:00:42.157793 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:00:42.157797 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:00:42.157800 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:00:42.158185 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:00:42.158403 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0807 23:00:42.158839 (+ 436us) server_negotiation.cc:299] Negotiation successful
0807 23:00:42.158998 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":423,"spinlock_wait_cycles":3968,"thread_start_us":270,"threads_started":1}
I20260807 23:00:42.160806 21026 heartbeater.cc:347] Connected to a master server at 127.25.94.62:40547
I20260807 23:00:42.161238 21026 heartbeater.cc:464] Registering TS with master...
I20260807 23:00:42.162138 21026 heartbeater.cc:511] Master 127.25.94.62:40547 requested a full tablet report, sending...
I20260807 23:00:42.164039 20235 ts_manager.cc:194] Registered new tserver with Master: a49447d7b0024e12b538ea065f867cb5 (127.25.94.3:41199)
I20260807 23:00:42.164988 20235 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.94.3@KRBTEST.COM'} at 127.25.94.3:51535
I20260807 23:00:42.175813 25976 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 23:00:42.183673 21032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:00:42.176906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48836 (local address 127.25.94.62:40547)
0807 23:00:42.177253 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0807 23:00:42.177263 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 23:00:42.177878 (+ 615us) server_negotiation.cc:407] Connection header received
0807 23:00:42.177988 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:00:42.177993 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:00:42.178194 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:00:42.178352 (+ 158us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 23:00:42.178895 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:00:42.179800 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:00:42.181546 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:00:42.181999 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:00:42.182101 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:00:42.182768 (+ 667us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 23:00:42.183059 (+ 291us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0807 23:00:42.183420 (+ 361us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48836: token expired
Metrics: {"server-negotiator.queue_time_us":164}
W20260807 23:00:42.183792 21032 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48836: token expired
W20260807 23:00:42.184667 20093 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.25.94.62:40547)
I20260807 23:00:42.197834 20546 heartbeater.cc:503] Master 127.25.94.62:40547 was elected leader, sending a full tablet report...
I20260807 23:00:42.199206 21032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:00:42.185326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48842 (local address 127.25.94.62:40547)
0807 23:00:42.185492 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0807 23:00:42.185496 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 23:00:42.185513 (+ 17us) server_negotiation.cc:407] Connection header received
0807 23:00:42.185631 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:00:42.185635 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:00:42.185696 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:00:42.185784 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:00:42.187231 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:00:42.187966 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:00:42.189329 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:00:42.189759 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:00:42.189871 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:00:42.192627 (+ 2756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:00:42.192632 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:00:42.192635 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 23:00:42.192671 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:00:42.195644 (+ 2973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:00:42.196500 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:00:42.196504 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:00:42.196506 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:00:42.196642 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:00:42.196997 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:00:42.197000 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:00:42.197002 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:00:42.197599 (+ 597us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:00:42.197867 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0807 23:00:42.198423 (+ 556us) server_negotiation.cc:299] Negotiation successful
0807 23:00:42.198716 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260807 23:00:42.203230 20238 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48842:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 23:00:42.211139 20238 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 23:00:42.248250 21048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:00:42.227706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.1:45459 (local address 127.0.0.1:59174)
0807 23:00:42.239152 (+ 11446us) negotiation.cc:107] Waiting for socket to connect
0807 23:00:42.239170 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 23:00:42.239311 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 23:00:42.240200 (+ 889us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 23:00:42.240205 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 23:00:42.240230 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 23:00:42.240682 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:00:42.240697 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:00:42.242625 (+ 1928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:00:42.242636 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:00:42.244747 (+ 2111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:00:42.244761 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:00:42.247031 (+ 2270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:00:42.247041 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:00:42.247326 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:00:42.247403 (+ 77us) client_negotiation.cc:769] Sending connection context
0807 23:00:42.247562 (+ 159us) client_negotiation.cc:240] Negotiation successful
0807 23:00:42.247773 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11197,"mutex_wait_us":73,"thread_start_us":10202,"threads_started":1}
I20260807 23:00:42.248555 21050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:00:42.224710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36910 (local address 127.25.94.3:41199)
0807 23:00:42.238770 (+ 14060us) server_negotiation.cc:206] Beginning negotiation
0807 23:00:42.238777 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 23:00:42.238813 (+ 36us) server_negotiation.cc:407] Connection header received
0807 23:00:42.238918 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:00:42.238929 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:00:42.239132 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:00:42.239348 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 23:00:42.240249 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:00:42.241639 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:00:42.245241 (+ 3602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:00:42.246784 (+ 1543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:00:42.246917 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:00:42.247144 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0807 23:00:42.247993 (+ 849us) server_negotiation.cc:299] Negotiation successful
0807 23:00:42.248179 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13895,"thread_start_us":13711,"threads_started":1}
I20260807 23:00:42.249078 21053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:00:42.228893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59174 (local address 127.25.94.1:45459)
0807 23:00:42.238113 (+ 9220us) server_negotiation.cc:206] Beginning negotiation
0807 23:00:42.238125 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 23:00:42.239443 (+ 1318us) server_negotiation.cc:407] Connection header received
0807 23:00:42.239542 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:00:42.239549 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:00:42.239767 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:00:42.239968 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 23:00:42.240878 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:00:42.242443 (+ 1565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:00:42.244929 (+ 2486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:00:42.246380 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:00:42.246542 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:00:42.246843 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0807 23:00:42.248673 (+ 1830us) server_negotiation.cc:299] Negotiation successful
0807 23:00:42.248841 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9037,"thread_start_us":8903,"threads_started":1}
I20260807 23:00:42.249142 21051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:00:42.225102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57580 (local address 127.25.94.2:36939)
0807 23:00:42.236593 (+ 11491us) server_negotiation.cc:206] Beginning negotiation
0807 23:00:42.236606 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0807 23:00:42.238217 (+ 1611us) server_negotiation.cc:407] Connection header received
0807 23:00:42.238502 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:00:42.238513 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:00:42.238776 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:00:42.239007 (+ 231us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 23:00:42.240173 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:00:42.241595 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:00:42.245241 (+ 3646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:00:42.246774 (+ 1533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:00:42.246917 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:00:42.247118 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0807 23:00:42.248618 (+ 1500us) server_negotiation.cc:299] Negotiation successful
0807 23:00:42.248806 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11301,"thread_start_us":11136,"threads_started":1}
I20260807 23:00:42.248472 21052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:00:42.224797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.2:36939 (local address 127.0.0.1:57580)
0807 23:00:42.238083 (+ 13286us) negotiation.cc:107] Waiting for socket to connect
0807 23:00:42.238103 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 23:00:42.238240 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 23:00:42.239336 (+ 1096us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 23:00:42.239346 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 23:00:42.239376 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 23:00:42.239920 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:00:42.239932 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:00:42.241753 (+ 1821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:00:42.241759 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:00:42.244988 (+ 3229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:00:42.244998 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:00:42.247387 (+ 2389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:00:42.247393 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:00:42.247687 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:00:42.247779 (+ 92us) client_negotiation.cc:769] Sending connection context
0807 23:00:42.247864 (+ 85us) client_negotiation.cc:240] Negotiation successful
0807 23:00:42.248019 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13147,"mutex_wait_us":404,"thread_start_us":14081,"threads_started":1}
I20260807 23:00:42.249222 21047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:00:42.224477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.3:41199 (local address 127.0.0.1:36910)
0807 23:00:42.237135 (+ 12658us) negotiation.cc:107] Waiting for socket to connect
0807 23:00:42.237230 (+ 95us) client_negotiation.cc:174] Beginning negotiation
0807 23:00:42.237457 (+ 227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 23:00:42.239659 (+ 2202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 23:00:42.239667 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 23:00:42.239699 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 23:00:42.240088 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:00:42.240099 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:00:42.241802 (+ 1703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:00:42.241808 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:00:42.244981 (+ 3173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:00:42.244998 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:00:42.247371 (+ 2373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:00:42.247378 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:00:42.247682 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:00:42.247773 (+ 91us) client_negotiation.cc:769] Sending connection context
0807 23:00:42.248716 (+ 943us) client_negotiation.cc:240] Negotiation successful
0807 23:00:42.248854 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12395,"mutex_wait_us":456,"thread_start_us":12217,"threads_started":1}
I20260807 23:00:42.253501 20654 tablet_service.cc:1467] Processing CreateTablet for tablet 6930c65beb564053afb6f702b33a0d2a (DEFAULT_TABLE table=test-table [id=7c6969adc14c44a6ac883bf2beac4be7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 23:00:42.253638 20441 tablet_service.cc:1467] Processing CreateTablet for tablet 6930c65beb564053afb6f702b33a0d2a (DEFAULT_TABLE table=test-table [id=7c6969adc14c44a6ac883bf2beac4be7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 23:00:42.253638 20909 tablet_service.cc:1467] Processing CreateTablet for tablet 6930c65beb564053afb6f702b33a0d2a (DEFAULT_TABLE table=test-table [id=7c6969adc14c44a6ac883bf2beac4be7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 23:00:42.255019 20654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6930c65beb564053afb6f702b33a0d2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 23:00:42.255195 20909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6930c65beb564053afb6f702b33a0d2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 23:00:42.255209 20441 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6930c65beb564053afb6f702b33a0d2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 23:00:42.268294 21057 tablet_bootstrap.cc:492] T 6930c65beb564053afb6f702b33a0d2a P 490f1fd977fb4fd586aed08107d9f266: Bootstrap starting.
I20260807 23:00:42.271452 21064 tablet_bootstrap.cc:492] T 6930c65beb564053afb6f702b33a0d2a P a49447d7b0024e12b538ea065f867cb5: Bootstrap starting.
I20260807 23:00:42.273296 21062 tablet_bootstrap.cc:492] T 6930c65beb564053afb6f702b33a0d2a P 8a9f21049c9c4db48d48db896dcce03d: Bootstrap starting.
I20260807 23:00:42.273478 21057 tablet_bootstrap.cc:654] T 6930c65beb564053afb6f702b33a0d2a P 490f1fd977fb4fd586aed08107d9f266: Neither blocks nor log segments found. Creating new log.
I20260807 23:00:42.274937 21057 log.cc:824] T 6930c65beb564053afb6f702b33a0d2a P 490f1fd977fb4fd586aed08107d9f266: Log is configured to *not* fsync() on all Append() calls
I20260807 23:00:42.275087 21064 tablet_bootstrap.cc:654] T 6930c65beb564053afb6f702b33a0d2a P a49447d7b0024e12b538ea065f867cb5: Neither blocks nor log segments found. Creating new log.
I20260807 23:00:42.276476 21064 log.cc:824] T 6930c65beb564053afb6f702b33a0d2a P a49447d7b0024e12b538ea065f867cb5: Log is configured to *not* fsync() on all Append() calls
I20260807 23:00:42.276515 21062 tablet_bootstrap.cc:654] T 6930c65beb564053afb6f702b33a0d2a P 8a9f21049c9c4db48d48db896dcce03d: Neither blocks nor log segments found. Creating new log.
I20260807 23:00:42.277937 21062 log.cc:824] T 6930c65beb564053afb6f702b33a0d2a P 8a9f21049c9c4db48d48db896dcce03d: Log is configured to *not* fsync() on all Append() calls
I20260807 23:00:42.279892 21057 tablet_bootstrap.cc:492] T 6930c65beb564053afb6f702b33a0d2a P 490f1fd977fb4fd586aed08107d9f266: No bootstrap required, opened a new log
I20260807 23:00:42.280406 21057 ts_tablet_manager.cc:1397] T 6930c65beb564053afb6f702b33a0d2a P 490f1fd977fb4fd586aed08107d9f266: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.007s
I20260807 23:00:42.284262 21064 tablet_bootstrap.cc:492] T 6930c65beb564053afb6f702b33a0d2a P a49447d7b0024e12b538ea065f867cb5: No bootstrap required, opened a new log
I20260807 23:00:42.284260 21062 tablet_bootstrap.cc:492] T 6930c65beb564053afb6f702b33a0d2a P 8a9f21049c9c4db48d48db896dcce03d: No bootstrap required, opened a new log
I20260807 23:00:42.284556 21062 ts_tablet_manager.cc:1397] T 6930c65beb564053afb6f702b33a0d2a P 8a9f21049c9c4db48d48db896dcce03d: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.003s
I20260807 23:00:42.284556 21064 ts_tablet_manager.cc:1397] T 6930c65beb564053afb6f702b33a0d2a P a49447d7b0024e12b538ea065f867cb5: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.004s
I20260807 23:00:42.289916 21057 raft_consensus.cc:348] T 6930c65beb564053afb6f702b33a0d2a P 490f1fd977fb4fd586aed08107d9f266 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a49447d7b0024e12b538ea065f867cb5" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 41199 } } peers { permanent_uuid: "490f1fd977fb4fd586aed08107d9f266" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 36939 } } peers { permanent_uuid: "8a9f21049c9c4db48d48db896dcce03d" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 45459 } }
I20260807 23:00:42.290369 21057 raft_consensus.cc:374] T 6930c65beb564053afb6f702b33a0d2a P 490f1fd977fb4fd586aed08107d9f266 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 23:00:42.290485 21057 raft_consensus.cc:729] T 6930c65beb564053afb6f702b33a0d2a P 490f1fd977fb4fd586aed08107d9f266 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 490f1fd977fb4fd586aed08107d9f266, State: Initialized, Role: FOLLOWER
I20260807 23:00:42.291160 21057 consensus_queue.cc:260] T 6930c65beb564053afb6f702b33a0d2a P 490f1fd977fb4fd586aed08107d9f266 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a49447d7b0024e12b538ea065f867cb5" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 41199 } } peers { permanent_uuid: "490f1fd977fb4fd586aed08107d9f266" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 36939 } } peers { permanent_uuid: "8a9f21049c9c4db48d48db896dcce03d" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 45459 } }
I20260807 23:00:42.292413 20781 heartbeater.cc:503] Master 127.25.94.62:40547 was elected leader, sending a full tablet report...
I20260807 23:00:42.293157 21057 ts_tablet_manager.cc:1428] T 6930c65beb564053afb6f702b33a0d2a P 490f1fd977fb4fd586aed08107d9f266: Time spent starting tablet: real 0.013s user 0.011s sys 0.001s
I20260807 23:00:42.296204 21062 raft_consensus.cc:348] T 6930c65beb564053afb6f702b33a0d2a P 8a9f21049c9c4db48d48db896dcce03d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a49447d7b0024e12b538ea065f867cb5" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 41199 } } peers { permanent_uuid: "490f1fd977fb4fd586aed08107d9f266" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 36939 } } peers { permanent_uuid: "8a9f21049c9c4db48d48db896dcce03d" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 45459 } }
I20260807 23:00:42.296204 21064 raft_consensus.cc:348] T 6930c65beb564053afb6f702b33a0d2a P a49447d7b0024e12b538ea065f867cb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a49447d7b0024e12b538ea065f867cb5" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 41199 } } peers { permanent_uuid: "490f1fd977fb4fd586aed08107d9f266" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 36939 } } peers { permanent_uuid: "8a9f21049c9c4db48d48db896dcce03d" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 45459 } }
I20260807 23:00:42.296733 21064 raft_consensus.cc:374] T 6930c65beb564053afb6f702b33a0d2a P a49447d7b0024e12b538ea065f867cb5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 23:00:42.296725 21062 raft_consensus.cc:374] T 6930c65beb564053afb6f702b33a0d2a P 8a9f21049c9c4db48d48db896dcce03d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 23:00:42.296851 21064 raft_consensus.cc:729] T 6930c65beb564053afb6f702b33a0d2a P a49447d7b0024e12b538ea065f867cb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a49447d7b0024e12b538ea065f867cb5, State: Initialized, Role: FOLLOWER
I20260807 23:00:42.296849 21062 raft_consensus.cc:729] T 6930c65beb564053afb6f702b33a0d2a P 8a9f21049c9c4db48d48db896dcce03d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9f21049c9c4db48d48db896dcce03d, State: Initialized, Role: FOLLOWER
I20260807 23:00:42.297610 21064 consensus_queue.cc:260] T 6930c65beb564053afb6f702b33a0d2a P a49447d7b0024e12b538ea065f867cb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a49447d7b0024e12b538ea065f867cb5" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 41199 } } peers { permanent_uuid: "490f1fd977fb4fd586aed08107d9f266" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 36939 } } peers { permanent_uuid: "8a9f21049c9c4db48d48db896dcce03d" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 45459 } }
I20260807 23:00:42.297610 21062 consensus_queue.cc:260] T 6930c65beb564053afb6f702b33a0d2a P 8a9f21049c9c4db48d48db896dcce03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a49447d7b0024e12b538ea065f867cb5" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 41199 } } peers { permanent_uuid: "490f1fd977fb4fd586aed08107d9f266" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 36939 } } peers { permanent_uuid: "8a9f21049c9c4db48d48db896dcce03d" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 45459 } }
I20260807 23:00:42.299060 21026 heartbeater.cc:503] Master 127.25.94.62:40547 was elected leader, sending a full tablet report...
I20260807 23:00:42.300026 21064 ts_tablet_manager.cc:1428] T 6930c65beb564053afb6f702b33a0d2a P a49447d7b0024e12b538ea065f867cb5: Time spent starting tablet: real 0.015s user 0.012s sys 0.004s
I20260807 23:00:42.300554 21062 ts_tablet_manager.cc:1428] T 6930c65beb564053afb6f702b33a0d2a P 8a9f21049c9c4db48d48db896dcce03d: Time spent starting tablet: real 0.016s user 0.015s sys 0.001s
W20260807 23:00:42.384438 21028 tablet.cc:2369] T 6930c65beb564053afb6f702b33a0d2a P a49447d7b0024e12b538ea065f867cb5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 23:00:42.390899 20785 tablet.cc:2369] T 6930c65beb564053afb6f702b33a0d2a P 490f1fd977fb4fd586aed08107d9f266: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 23:00:42.402339 20547 tablet.cc:2369] T 6930c65beb564053afb6f702b33a0d2a P 8a9f21049c9c4db48d48db896dcce03d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 23:00:42.417268 21079 raft_consensus.cc:482] T 6930c65beb564053afb6f702b33a0d2a P 8a9f21049c9c4db48d48db896dcce03d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 23:00:42.417631 21079 raft_consensus.cc:504] T 6930c65beb564053afb6f702b33a0d2a P 8a9f21049c9c4db48d48db896dcce03d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a49447d7b0024e12b538ea065f867cb5" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 41199 } } peers { permanent_uuid: "490f1fd977fb4fd586aed08107d9f266" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 36939 } } peers { permanent_uuid: "8a9f21049c9c4db48d48db896dcce03d" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 45459 } }
I20260807 23:00:42.419430 21079 leader_election.cc:288] T 6930c65beb564053afb6f702b33a0d2a P 8a9f21049c9c4db48d48db896dcce03d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a49447d7b0024e12b538ea065f867cb5 (127.25.94.3:41199), 490f1fd977fb4fd586aed08107d9f266 (127.25.94.2:36939)
I20260807 23:00:42.426326 21050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:00:42.419897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.1:51477 (local address 127.25.94.3:41199)
0807 23:00:42.420115 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0807 23:00:42.420122 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 23:00:42.420738 (+ 616us) server_negotiation.cc:407] Connection header received
0807 23:00:42.421003 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:00:42.421007 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:00:42.421070 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:00:42.421201 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 23:00:42.421803 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:00:42.422545 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:00:42.424442 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:00:42.425314 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:00:42.425462 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:00:42.425582 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0807 23:00:42.425905 (+ 323us) server_negotiation.cc:299] Negotiation successful
0807 23:00:42.426040 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260807 23:00:42.426447 21051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:00:42.419894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.1:51865 (local address 127.25.94.2:36939)
0807 23:00:42.420115 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0807 23:00:42.420122 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 23:00:42.421213 (+ 1091us) server_negotiation.cc:407] Connection header received
0807 23:00:42.421441 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:00:42.421446 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:00:42.421507 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:00:42.421607 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 23:00:42.422153 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:00:42.422850 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:00:42.424660 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:00:42.425499 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:00:42.425657 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:00:42.425766 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0807 23:00:42.426051 (+ 285us) server_negotiation.cc:299] Negotiation successful
0807 23:00:42.426177 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260807 23:00:42.426528 21082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:00:42.419771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.3:41199 (local address 127.25.94.1:51477)
0807 23:00:42.420591 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0807 23:00:42.420619 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0807 23:00:42.420823 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 23:00:42.421255 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 23:00:42.421262 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 23:00:42.421298 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 23:00:42.421640 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:00:42.421656 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:00:42.422697 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:00:42.422702 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:00:42.424272 (+ 1570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:00:42.424285 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:00:42.425543 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:00:42.425549 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:00:42.425731 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:00:42.425755 (+ 24us) client_negotiation.cc:769] Sending connection context
0807 23:00:42.425879 (+ 124us) client_negotiation.cc:240] Negotiation successful
0807 23:00:42.426011 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":707,"mutex_wait_us":6,"thread_start_us":524,"threads_started":1}
I20260807 23:00:42.426611 21083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:00:42.419771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.2:36939 (local address 127.25.94.1:51865)
0807 23:00:42.421080 (+ 1309us) negotiation.cc:107] Waiting for socket to connect
0807 23:00:42.421099 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 23:00:42.421290 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 23:00:42.421636 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 23:00:42.421640 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 23:00:42.421664 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 23:00:42.422006 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:00:42.422015 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:00:42.423000 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:00:42.423008 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:00:42.424503 (+ 1495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:00:42.424515 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:00:42.425741 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:00:42.425747 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:00:42.425897 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:00:42.425919 (+ 22us) client_negotiation.cc:769] Sending connection context
0807 23:00:42.426026 (+ 107us) client_negotiation.cc:240] Negotiation successful
0807 23:00:42.426161 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1214,"mutex_wait_us":60,"thread_start_us":1018,"threads_started":1}
I20260807 23:00:42.427682 20688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6930c65beb564053afb6f702b33a0d2a" candidate_uuid: "8a9f21049c9c4db48d48db896dcce03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "490f1fd977fb4fd586aed08107d9f266" is_pre_election: true
I20260807 23:00:42.427682 20942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6930c65beb564053afb6f702b33a0d2a" candidate_uuid: "8a9f21049c9c4db48d48db896dcce03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a49447d7b0024e12b538ea065f867cb5" is_pre_election: true
I20260807 23:00:42.428292 20688 raft_consensus.cc:2454] T 6930c65beb564053afb6f702b33a0d2a P 490f1fd977fb4fd586aed08107d9f266 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a9f21049c9c4db48d48db896dcce03d in term 0.
I20260807 23:00:42.428292 20942 raft_consensus.cc:2454] T 6930c65beb564053afb6f702b33a0d2a P a49447d7b0024e12b538ea065f867cb5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a9f21049c9c4db48d48db896dcce03d in term 0.
I20260807 23:00:42.429057 20366 leader_election.cc:302] T 6930c65beb564053afb6f702b33a0d2a P 8a9f21049c9c4db48d48db896dcce03d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 490f1fd977fb4fd586aed08107d9f266, 8a9f21049c9c4db48d48db896dcce03d; no voters:
I20260807 23:00:42.429533 21079 raft_consensus.cc:2781] T 6930c65beb564053afb6f702b33a0d2a P 8a9f21049c9c4db48d48db896dcce03d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 23:00:42.429720 21079 raft_consensus.cc:482] T 6930c65beb564053afb6f702b33a0d2a P 8a9f21049c9c4db48d48db896dcce03d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 23:00:42.429836 21079 raft_consensus.cc:3037] T 6930c65beb564053afb6f702b33a0d2a P 8a9f21049c9c4db48d48db896dcce03d [term 0 FOLLOWER]: Advancing to term 1
I20260807 23:00:42.432109 21079 raft_consensus.cc:504] T 6930c65beb564053afb6f702b33a0d2a P 8a9f21049c9c4db48d48db896dcce03d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a49447d7b0024e12b538ea065f867cb5" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 41199 } } peers { permanent_uuid: "490f1fd977fb4fd586aed08107d9f266" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 36939 } } peers { permanent_uuid: "8a9f21049c9c4db48d48db896dcce03d" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 45459 } }
I20260807 23:00:42.432816 21079 leader_election.cc:288] T 6930c65beb564053afb6f702b33a0d2a P 8a9f21049c9c4db48d48db896dcce03d [CANDIDATE]: Term 1 election: Requested vote from peers a49447d7b0024e12b538ea065f867cb5 (127.25.94.3:41199), 490f1fd977fb4fd586aed08107d9f266 (127.25.94.2:36939)
I20260807 23:00:42.433410 20688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6930c65beb564053afb6f702b33a0d2a" candidate_uuid: "8a9f21049c9c4db48d48db896dcce03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "490f1fd977fb4fd586aed08107d9f266"
I20260807 23:00:42.433404 20942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6930c65beb564053afb6f702b33a0d2a" candidate_uuid: "8a9f21049c9c4db48d48db896dcce03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a49447d7b0024e12b538ea065f867cb5"
I20260807 23:00:42.433634 20942 raft_consensus.cc:3037] T 6930c65beb564053afb6f702b33a0d2a P a49447d7b0024e12b538ea065f867cb5 [term 0 FOLLOWER]: Advancing to term 1
I20260807 23:00:42.433638 20688 raft_consensus.cc:3037] T 6930c65beb564053afb6f702b33a0d2a P 490f1fd977fb4fd586aed08107d9f266 [term 0 FOLLOWER]: Advancing to term 1
I20260807 23:00:42.437023 20942 raft_consensus.cc:2454] T 6930c65beb564053afb6f702b33a0d2a P a49447d7b0024e12b538ea065f867cb5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a9f21049c9c4db48d48db896dcce03d in term 1.
I20260807 23:00:42.437023 20688 raft_consensus.cc:2454] T 6930c65beb564053afb6f702b33a0d2a P 490f1fd977fb4fd586aed08107d9f266 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a9f21049c9c4db48d48db896dcce03d in term 1.
I20260807 23:00:42.437539 20363 leader_election.cc:302] T 6930c65beb564053afb6f702b33a0d2a P 8a9f21049c9c4db48d48db896dcce03d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 490f1fd977fb4fd586aed08107d9f266, 8a9f21049c9c4db48d48db896dcce03d, a49447d7b0024e12b538ea065f867cb5; no voters:
I20260807 23:00:42.437896 21079 raft_consensus.cc:2781] T 6930c65beb564053afb6f702b33a0d2a P 8a9f21049c9c4db48d48db896dcce03d [term 1 FOLLOWER]: Leader election won for term 1
I20260807 23:00:42.438359 21079 raft_consensus.cc:686] T 6930c65beb564053afb6f702b33a0d2a P 8a9f21049c9c4db48d48db896dcce03d [term 1 LEADER]: Becoming Leader. State: Replica: 8a9f21049c9c4db48d48db896dcce03d, State: Running, Role: LEADER
I20260807 23:00:42.438876 21079 consensus_queue.cc:237] T 6930c65beb564053afb6f702b33a0d2a P 8a9f21049c9c4db48d48db896dcce03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a49447d7b0024e12b538ea065f867cb5" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 41199 } } peers { permanent_uuid: "490f1fd977fb4fd586aed08107d9f266" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 36939 } } peers { permanent_uuid: "8a9f21049c9c4db48d48db896dcce03d" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 45459 } }
I20260807 23:00:42.443750 20238 catalog_manager.cc:5495] T 6930c65beb564053afb6f702b33a0d2a P 8a9f21049c9c4db48d48db896dcce03d reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a9f21049c9c4db48d48db896dcce03d (127.25.94.1). New cstate: current_term: 1 leader_uuid: "8a9f21049c9c4db48d48db896dcce03d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a49447d7b0024e12b538ea065f867cb5" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 41199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "490f1fd977fb4fd586aed08107d9f266" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 36939 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a9f21049c9c4db48d48db896dcce03d" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 45459 } health_report { overall_health: HEALTHY } } }
I20260807 23:00:42.499888 21053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:00:42.493957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38438 (local address 127.25.94.1:45459)
0807 23:00:42.494190 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0807 23:00:42.494197 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 23:00:42.494298 (+ 101us) server_negotiation.cc:407] Connection header received
0807 23:00:42.494530 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:00:42.494535 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:00:42.494627 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:00:42.494805 (+ 178us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 23:00:42.495191 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:00:42.495965 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:00:42.497484 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:00:42.497967 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:00:42.498112 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:00:42.498709 (+ 597us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 23:00:42.499283 (+ 574us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 23:00:42.499438 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0807 23:00:42.499524 (+ 86us) server_negotiation.cc:299] Negotiation successful
0807 23:00:42.499608 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260807 23:00:42.508664 20688 raft_consensus.cc:1261] T 6930c65beb564053afb6f702b33a0d2a P 490f1fd977fb4fd586aed08107d9f266 [term 1 FOLLOWER]: Refusing update from remote peer 8a9f21049c9c4db48d48db896dcce03d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 23:00:42.508674 20942 raft_consensus.cc:1261] T 6930c65beb564053afb6f702b33a0d2a P a49447d7b0024e12b538ea065f867cb5 [term 1 FOLLOWER]: Refusing update from remote peer 8a9f21049c9c4db48d48db896dcce03d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 23:00:42.509809 21084 consensus_queue.cc:1035] T 6930c65beb564053afb6f702b33a0d2a P 8a9f21049c9c4db48d48db896dcce03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a49447d7b0024e12b538ea065f867cb5" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 41199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 23:00:42.510181 21079 consensus_queue.cc:1035] T 6930c65beb564053afb6f702b33a0d2a P 8a9f21049c9c4db48d48db896dcce03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "490f1fd977fb4fd586aed08107d9f266" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 36939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 23:00:42.523702 21090 mvcc.cc:205] Tried to move back new op lower bound from 7316044359703793664 to 7316044359433719808. Current Snapshot: MvccSnapshot[applied={T|T < 7316044359703793664}]
I20260807 23:00:42.534727 20235 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48842:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 23:00:42.534993 20235 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48842:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 23:00:42.538432 20235 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7cf6aa251b54475d819168a882c99955: Sending DeleteTablet for 3 replicas of tablet 6930c65beb564053afb6f702b33a0d2a
I20260807 23:00:42.539698 20441 tablet_service.cc:1514] Processing DeleteTablet for tablet 6930c65beb564053afb6f702b33a0d2a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 23:00:42 UTC) from {username='kudu', principal='kudu/127.25.94.62@KRBTEST.COM'} at 127.0.0.1:59174
I20260807 23:00:42.539677 20654 tablet_service.cc:1514] Processing DeleteTablet for tablet 6930c65beb564053afb6f702b33a0d2a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 23:00:42 UTC) from {username='kudu', principal='kudu/127.25.94.62@KRBTEST.COM'} at 127.0.0.1:57580
I20260807 23:00:42.539700 20909 tablet_service.cc:1514] Processing DeleteTablet for tablet 6930c65beb564053afb6f702b33a0d2a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 23:00:42 UTC) from {username='kudu', principal='kudu/127.25.94.62@KRBTEST.COM'} at 127.0.0.1:36910
I20260807 23:00:42.541553 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 20296
I20260807 23:00:42.552600 21111 tablet_replica.cc:330] T 6930c65beb564053afb6f702b33a0d2a P a49447d7b0024e12b538ea065f867cb5: stopping tablet replica
I20260807 23:00:42.553647 21111 raft_consensus.cc:2229] T 6930c65beb564053afb6f702b33a0d2a P a49447d7b0024e12b538ea065f867cb5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 23:00:42.553779 21115 tablet_replica.cc:330] T 6930c65beb564053afb6f702b33a0d2a P 490f1fd977fb4fd586aed08107d9f266: stopping tablet replica
I20260807 23:00:42.554267 21115 raft_consensus.cc:2229] T 6930c65beb564053afb6f702b33a0d2a P 490f1fd977fb4fd586aed08107d9f266 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 23:00:42.554941 21115 raft_consensus.cc:2258] T 6930c65beb564053afb6f702b33a0d2a P 490f1fd977fb4fd586aed08107d9f266 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 23:00:42.555953 21112 tablet_replica.cc:330] T 6930c65beb564053afb6f702b33a0d2a P 8a9f21049c9c4db48d48db896dcce03d: stopping tablet replica
I20260807 23:00:42.556565 21112 raft_consensus.cc:2229] T 6930c65beb564053afb6f702b33a0d2a P 8a9f21049c9c4db48d48db896dcce03d [term 1 LEADER]: Raft consensus shutting down.
I20260807 23:00:42.557399 21112 raft_consensus.cc:2258] T 6930c65beb564053afb6f702b33a0d2a P 8a9f21049c9c4db48d48db896dcce03d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 23:00:42.558594 21111 raft_consensus.cc:2258] T 6930c65beb564053afb6f702b33a0d2a P a49447d7b0024e12b538ea065f867cb5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 23:00:42.559929 21115 ts_tablet_manager.cc:1905] T 6930c65beb564053afb6f702b33a0d2a P 490f1fd977fb4fd586aed08107d9f266: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 23:00:42.561137 21111 ts_tablet_manager.cc:1905] T 6930c65beb564053afb6f702b33a0d2a P a49447d7b0024e12b538ea065f867cb5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 23:00:42.564641 21112 ts_tablet_manager.cc:1905] T 6930c65beb564053afb6f702b33a0d2a P 8a9f21049c9c4db48d48db896dcce03d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 23:00:42.568045 21053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:00:42.543720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38448 (local address 127.25.94.1:45459)
0807 23:00:42.543958 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0807 23:00:42.543966 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 23:00:42.543991 (+ 25us) server_negotiation.cc:407] Connection header received
0807 23:00:42.544068 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:00:42.544076 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:00:42.544186 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:00:42.544312 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:00:42.547654 (+ 3342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:00:42.549388 (+ 1734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:00:42.552133 (+ 2745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:00:42.553203 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:00:42.553351 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:00:42.558843 (+ 5492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:00:42.558859 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:00:42.558878 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 23:00:42.558940 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:00:42.563286 (+ 4346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:00:42.564392 (+ 1106us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:00:42.564402 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:00:42.564408 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:00:42.564602 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:00:42.565121 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:00:42.565128 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:00:42.565132 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:00:42.565648 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:00:42.565959 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0807 23:00:42.567326 (+ 1367us) server_negotiation.cc:299] Negotiation successful
0807 23:00:42.567647 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260807 23:00:42.568809 20531 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 23:00:42.569041 21111 ts_tablet_manager.cc:1918] T 6930c65beb564053afb6f702b33a0d2a P a49447d7b0024e12b538ea065f867cb5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 23:00:42.569337 21111 log.cc:1196] T 6930c65beb564053afb6f702b33a0d2a P a49447d7b0024e12b538ea065f867cb5: Deleting WAL directory at /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-2/wal/wals/6930c65beb564053afb6f702b33a0d2a
I20260807 23:00:42.569975 21111 ts_tablet_manager.cc:1939] T 6930c65beb564053afb6f702b33a0d2a P a49447d7b0024e12b538ea065f867cb5: Deleting consensus metadata
I20260807 23:00:42.571835 21115 ts_tablet_manager.cc:1918] T 6930c65beb564053afb6f702b33a0d2a P 490f1fd977fb4fd586aed08107d9f266: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 23:00:42.572041 21115 log.cc:1196] T 6930c65beb564053afb6f702b33a0d2a P 490f1fd977fb4fd586aed08107d9f266: Deleting WAL directory at /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-1/wal/wals/6930c65beb564053afb6f702b33a0d2a
I20260807 23:00:42.572608 20208 catalog_manager.cc:4841] TS a49447d7b0024e12b538ea065f867cb5 (127.25.94.3:41199): tablet 6930c65beb564053afb6f702b33a0d2a (table test-table [id=7c6969adc14c44a6ac883bf2beac4be7]) successfully deleted
I20260807 23:00:42.572654 21115 ts_tablet_manager.cc:1939] T 6930c65beb564053afb6f702b33a0d2a P 490f1fd977fb4fd586aed08107d9f266: Deleting consensus metadata
I20260807 23:00:42.574684 20211 catalog_manager.cc:4841] TS 490f1fd977fb4fd586aed08107d9f266 (127.25.94.2:36939): tablet 6930c65beb564053afb6f702b33a0d2a (table test-table [id=7c6969adc14c44a6ac883bf2beac4be7]) successfully deleted
I20260807 23:00:42.671994 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 20296
I20260807 23:00:42.702301 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 20554
W20260807 23:00:42.702342 20209 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.25.94.1:45459 (error 108)
I20260807 23:00:42.704087 21047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:00:42.703649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.1:45459 (local address 127.0.0.1:38464)
0807 23:00:42.703819 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0807 23:00:42.703947 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.1:45459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260807 23:00:42.704352 20209 catalog_manager.cc:4584] TS 8a9f21049c9c4db48d48db896dcce03d (127.25.94.1:45459): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6930c65beb564053afb6f702b33a0d2a: Network error: Client connection negotiation failed: client connection to 127.25.94.1:45459: connect: Connection refused (error 111)
I20260807 23:00:42.716933 21051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:00:42.703383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41258 (local address 127.25.94.2:36939)
0807 23:00:42.703569 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0807 23:00:42.703573 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 23:00:42.703591 (+ 18us) server_negotiation.cc:407] Connection header received
0807 23:00:42.703646 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:00:42.703650 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:00:42.703708 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:00:42.703834 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:00:42.705645 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:00:42.706506 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:00:42.707884 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:00:42.708279 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:00:42.708414 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:00:42.711235 (+ 2821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:00:42.711249 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:00:42.711264 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 23:00:42.711301 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:00:42.714095 (+ 2794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:00:42.714808 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:00:42.714815 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:00:42.714817 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:00:42.714946 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:00:42.715326 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:00:42.715332 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:00:42.715335 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:00:42.715726 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:00:42.716024 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0807 23:00:42.716367 (+ 343us) server_negotiation.cc:299] Negotiation successful
0807 23:00:42.716584 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260807 23:00:42.717685 20769 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 23:00:42.757964 21047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:00:42.757519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.1:45459 (local address 127.0.0.1:38472)
0807 23:00:42.757724 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0807 23:00:42.757812 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.1:45459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260807 23:00:42.759028 21047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:00:42.758673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.1:45459 (local address 127.0.0.1:38480)
0807 23:00:42.758830 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0807 23:00:42.758911 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.1:45459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260807 23:00:42.809068 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 20554
I20260807 23:00:42.819919 21047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:00:42.819429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.1:45459 (local address 127.0.0.1:38488)
0807 23:00:42.819648 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0807 23:00:42.819736 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.1:45459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260807 23:00:42.821094 21047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:00:42.820734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.1:45459 (local address 127.0.0.1:38498)
0807 23:00:42.820908 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0807 23:00:42.820985 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.1:45459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260807 23:00:42.835155 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 20787
I20260807 23:00:42.848974 21050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:00:42.836016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51934 (local address 127.25.94.3:41199)
0807 23:00:42.836198 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0807 23:00:42.836202 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 23:00:42.836222 (+ 20us) server_negotiation.cc:407] Connection header received
0807 23:00:42.836364 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:00:42.836368 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:00:42.836442 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:00:42.836537 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:00:42.838078 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:00:42.838817 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:00:42.840152 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:00:42.840538 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:00:42.840635 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:00:42.843313 (+ 2678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:00:42.843327 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:00:42.843344 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 23:00:42.843380 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:00:42.846137 (+ 2757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:00:42.846999 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:00:42.847007 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:00:42.847009 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:00:42.847150 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:00:42.847488 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:00:42.847491 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:00:42.847493 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:00:42.847866 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:00:42.848111 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0807 23:00:42.848406 (+ 295us) server_negotiation.cc:299] Negotiation successful
0807 23:00:42.848567 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260807 23:00:42.849838 21016 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 23:00:42.901952 21047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:00:42.901422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.1:45459 (local address 127.0.0.1:38510)
0807 23:00:42.901657 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0807 23:00:42.901767 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.1:45459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260807 23:00:42.903069 21047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:00:42.902688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.1:45459 (local address 127.0.0.1:38522)
0807 23:00:42.902889 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0807 23:00:42.902967 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.1:45459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
I20260807 23:00:42.943748 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 20787
I20260807 23:00:42.968367 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 20179
I20260807 23:00:42.969826 20285 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 23:00:43.084991 21047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:00:43.084206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.1:45459 (local address 127.0.0.1:38532)
0807 23:00:43.084611 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0807 23:00:43.084727 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.1:45459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241}
I20260807 23:00:43.084972 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 20179
I20260807 23:01:03.116545 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.62:40547
--webserver_interface=127.25.94.62
--webserver_port=44809
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.62.keytab
--principal=kudu/127.25.94.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.20:35499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.94.62:40547
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 23:01:03.497442 21125 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 23:01:03.497776 21125 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 23:01:03.497874 21125 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 23:01:03.505312 21125 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260807 23:01:03.505424 21125 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260807 23:01:03.505463 21125 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260807 23:01:03.505517 21125 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 23:01:03.505570 21125 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 23:01:03.505631 21125 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 23:01:03.505676 21125 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 23:01:03.505723 21125 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 23:01:03.515852 21125 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:35499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.25.94.62:40547
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.62.keytab
--principal=kudu/127.25.94.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.62:40547
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.94.62
--webserver_port=44809
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 23:01:03.517211 21125 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 23:01:03.518550 21125 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 23:01:03.531955 21134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 23:01:03.531977 21135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 23:01:03.531951 21139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 23:01:03.533066 21125 server_base.cc:1034] running on GCE node
I20260807 23:01:03.534016 21125 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 23:01:03.535668 21125 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 23:01:03.537007 21125 hybrid_clock.cc:648] HybridClock initialized: now 1786143663536942 us; error 63 us; skew 500 ppm
I20260807 23:01:03.537451 21125 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 23:01:03 dist-test-slave-z8x9 krb5kdc[19243](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143663, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 23:01:03.544589 21125 init.cc:377] Logged in from keytab as kudu/127.25.94.62@KRBTEST.COM (short username kudu)
I20260807 23:01:03.547041 21125 webserver.cc:460] Webserver started at http://127.25.94.62:44809/ using document root <none> and password file <none>
I20260807 23:01:03.548190 21125 fs_manager.cc:359] Metadata directory not provided
I20260807 23:01:03.548321 21125 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 23:01:03.555608 21125 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260807 23:01:03.558864 21149 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 23:01:03.560496 21125 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260807 23:01:03.560647 21125 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/master-0/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/master-0/wal
uuid: "7cf6aa251b54475d819168a882c99955"
format_stamp: "Formatted at 2026-08-07 23:00:17 on dist-test-slave-z8x9"
server_key: "bc426fdb20e3d3a60185290f897b98e4"
server_key_iv: "388cc39b6271fbc10782fbce5d3e08ff"
server_key_version: "encryptionkey@0"
I20260807 23:01:03.561708 21125 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 23:01:03.579469 21125 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 23:01:03.584283 21125 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 23:01:03.584612 21125 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 23:01:03.605146 21125 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.62:40547
I20260807 23:01:03.605250 21241 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.62:40547 every 8 connection(s)
I20260807 23:01:03.606848 21125 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
I20260807 23:01:03.611433 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 21125
I20260807 23:01:03.612537 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.1:45459
--local_ip_for_outbound_sockets=127.25.94.1
--tserver_master_addrs=127.25.94.62:40547
--webserver_port=35209
--webserver_interface=127.25.94.1
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.1.keytab
--principal=kudu/127.25.94.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.20:35499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 23:01:03.615305 21242 sys_catalog.cc:263] Verifying existing consensus state
I20260807 23:01:03.620776 21242 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7cf6aa251b54475d819168a882c99955: Bootstrap starting.
I20260807 23:01:03.645157 21242 log.cc:824] T 00000000000000000000000000000000 P 7cf6aa251b54475d819168a882c99955: Log is configured to *not* fsync() on all Append() calls
I20260807 23:01:03.662305 21242 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7cf6aa251b54475d819168a882c99955: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 23:01:03.662945 21242 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7cf6aa251b54475d819168a882c99955: Bootstrap complete.
I20260807 23:01:03.673519 21242 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7cf6aa251b54475d819168a882c99955 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cf6aa251b54475d819168a882c99955" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 40547 } }
I20260807 23:01:03.674525 21242 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7cf6aa251b54475d819168a882c99955 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7cf6aa251b54475d819168a882c99955, State: Initialized, Role: FOLLOWER
I20260807 23:01:03.675334 21242 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7cf6aa251b54475d819168a882c99955 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cf6aa251b54475d819168a882c99955" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 40547 } }
I20260807 23:01:03.675590 21242 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7cf6aa251b54475d819168a882c99955 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 23:01:03.675712 21242 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7cf6aa251b54475d819168a882c99955 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 23:01:03.675918 21242 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7cf6aa251b54475d819168a882c99955 [term 2 FOLLOWER]: Advancing to term 3
I20260807 23:01:03.680286 21242 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7cf6aa251b54475d819168a882c99955 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cf6aa251b54475d819168a882c99955" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 40547 } }
I20260807 23:01:03.680835 21242 leader_election.cc:302] T 00000000000000000000000000000000 P 7cf6aa251b54475d819168a882c99955 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7cf6aa251b54475d819168a882c99955; no voters:
I20260807 23:01:03.681743 21242 leader_election.cc:288] T 00000000000000000000000000000000 P 7cf6aa251b54475d819168a882c99955 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260807 23:01:03.682042 21247 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7cf6aa251b54475d819168a882c99955 [term 3 FOLLOWER]: Leader election won for term 3
I20260807 23:01:03.683624 21247 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7cf6aa251b54475d819168a882c99955 [term 3 LEADER]: Becoming Leader. State: Replica: 7cf6aa251b54475d819168a882c99955, State: Running, Role: LEADER
I20260807 23:01:03.684573 21242 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7cf6aa251b54475d819168a882c99955 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 23:01:03.684401 21247 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7cf6aa251b54475d819168a882c99955 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cf6aa251b54475d819168a882c99955" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 40547 } }
I20260807 23:01:03.687178 21249 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7cf6aa251b54475d819168a882c99955 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7cf6aa251b54475d819168a882c99955. Latest consensus state: current_term: 3 leader_uuid: "7cf6aa251b54475d819168a882c99955" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cf6aa251b54475d819168a882c99955" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 40547 } } }
I20260807 23:01:03.687567 21249 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7cf6aa251b54475d819168a882c99955 [sys.catalog]: This master's current role is: LEADER
I20260807 23:01:03.688171 21248 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7cf6aa251b54475d819168a882c99955 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "7cf6aa251b54475d819168a882c99955" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cf6aa251b54475d819168a882c99955" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 40547 } } }
I20260807 23:01:03.688463 21248 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7cf6aa251b54475d819168a882c99955 [sys.catalog]: This master's current role is: LEADER
I20260807 23:01:03.690670 21254 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 23:01:03.702466 21254 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 23:01:03.704659 21254 catalog_manager.cc:1259] Loaded cluster ID: 7a21efc46e6c4d0689df4ef042c69b80
I20260807 23:01:03.704844 21254 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 23:01:03.710402 21254 catalog_manager.cc:1504] Loading token signing keys...
I20260807 23:01:03.712747 21254 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 7cf6aa251b54475d819168a882c99955: Loaded TSK: 0
I20260807 23:01:03.713052 21254 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 7cf6aa251b54475d819168a882c99955: Loaded TSK: 1
F20260807 23:01:03.714617 21254 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f10e6e869ad google::LogMessage::Fail() at ??:0
@ 0x7f10e6e8988a google::LogMessage::SendToLog() at ??:0
@ 0x7f10e6e864de google::LogMessage::Flush() at ??:0
@ 0x7f10e6e87a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f10eabea7d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f10eabd973e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f10fc8b51c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f10fc8b4783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f10fc9402a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f10fc9400c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f10fc98a5e9 std::function<>::operator()() at ??:0
@ 0x7f10fc8b8cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f10fc8b6c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f10fc93e610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f10fc93e439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f10e9ad2345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f10e9ad591c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f10e9ad5799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f10e9ab0eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f10ecd8a6db start_thread at ??:0
@ 0x7f10e5d2061f clone at ??:0
W20260807 23:01:04.033471 21244 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 23:01:04.033823 21244 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 23:01:04.033999 21244 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 23:01:04.041925 21244 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 23:01:04.042076 21244 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 23:01:04.042255 21244 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.1
I20260807 23:01:04.052026 21244 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:35499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.1.keytab
--principal=kudu/127.25.94.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.1:45459
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.94.1
--webserver_port=35209
--tserver_master_addrs=127.25.94.62:40547
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.1
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 23:01:04.053398 21244 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 23:01:04.054714 21244 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 23:01:04.068677 21324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 23:01:04.068634 21326 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 23:01:04.068704 21330 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 23:01:04.069957 21244 server_base.cc:1034] running on GCE node
I20260807 23:01:04.070669 21244 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 23:01:04.071696 21244 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 23:01:04.072957 21244 hybrid_clock.cc:648] HybridClock initialized: now 1786143664072915 us; error 54 us; skew 500 ppm
I20260807 23:01:04.073401 21244 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 23:01:04 dist-test-slave-z8x9 krb5kdc[19243](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143664, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 23:01:04.076713 21244 init.cc:377] Logged in from keytab as kudu/127.25.94.1@KRBTEST.COM (short username kudu)
I20260807 23:01:04.078442 21244 webserver.cc:460] Webserver started at http://127.25.94.1:35209/ using document root <none> and password file <none>
I20260807 23:01:04.079214 21244 fs_manager.cc:359] Metadata directory not provided
I20260807 23:01:04.079378 21244 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 23:01:04.086090 21244 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.003s
I20260807 23:01:04.089658 21340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 23:01:04.091193 21244 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260807 23:01:04.091426 21244 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-0/wal
uuid: "8a9f21049c9c4db48d48db896dcce03d"
format_stamp: "Formatted at 2026-08-07 23:00:17 on dist-test-slave-z8x9"
server_key: "fe42c621df0923528e732238bb21d9b0"
server_key_iv: "3420d0b1ab3d2891a30b640606475760"
server_key_version: "encryptionkey@0"
I20260807 23:01:04.092517 21244 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 23:01:04.140349 21244 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 23:01:04.146366 21244 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 23:01:04.146723 21244 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 23:01:04.147838 21244 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 23:01:04.150170 21350 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260807 23:01:04.156520 21244 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260807 23:01:04.156764 21244 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.007s user 0.000s sys 0.001s
I20260807 23:01:04.156955 21244 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260807 23:01:04.160485 21244 ts_tablet_manager.cc:610] Registered 1 tablets
I20260807 23:01:04.160633 21244 ts_tablet_manager.cc:589] Time spent register tablets: real 0.004s user 0.000s sys 0.001s
I20260807 23:01:04.161434 21350 tablet_bootstrap.cc:492] T 6930c65beb564053afb6f702b33a0d2a P 8a9f21049c9c4db48d48db896dcce03d: Bootstrap starting.
I20260807 23:01:04.171161 21350 log.cc:824] T 6930c65beb564053afb6f702b33a0d2a P 8a9f21049c9c4db48d48db896dcce03d: Log is configured to *not* fsync() on all Append() calls
I20260807 23:01:04.189153 21350 tablet_bootstrap.cc:492] T 6930c65beb564053afb6f702b33a0d2a P 8a9f21049c9c4db48d48db896dcce03d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 23:01:04.190140 21350 tablet_bootstrap.cc:492] T 6930c65beb564053afb6f702b33a0d2a P 8a9f21049c9c4db48d48db896dcce03d: Bootstrap complete.
I20260807 23:01:04.190891 21350 ts_tablet_manager.cc:1397] T 6930c65beb564053afb6f702b33a0d2a P 8a9f21049c9c4db48d48db896dcce03d: Time spent bootstrapping tablet: real 0.030s user 0.027s sys 0.000s
I20260807 23:01:04.202173 21350 raft_consensus.cc:348] T 6930c65beb564053afb6f702b33a0d2a P 8a9f21049c9c4db48d48db896dcce03d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a49447d7b0024e12b538ea065f867cb5" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 41199 } } peers { permanent_uuid: "490f1fd977fb4fd586aed08107d9f266" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 36939 } } peers { permanent_uuid: "8a9f21049c9c4db48d48db896dcce03d" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 45459 } }
I20260807 23:01:04.204068 21350 raft_consensus.cc:729] T 6930c65beb564053afb6f702b33a0d2a P 8a9f21049c9c4db48d48db896dcce03d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9f21049c9c4db48d48db896dcce03d, State: Initialized, Role: FOLLOWER
I20260807 23:01:04.205433 21350 consensus_queue.cc:260] T 6930c65beb564053afb6f702b33a0d2a P 8a9f21049c9c4db48d48db896dcce03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a49447d7b0024e12b538ea065f867cb5" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 41199 } } peers { permanent_uuid: "490f1fd977fb4fd586aed08107d9f266" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 36939 } } peers { permanent_uuid: "8a9f21049c9c4db48d48db896dcce03d" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 45459 } }
I20260807 23:01:04.207261 21350 ts_tablet_manager.cc:1428] T 6930c65beb564053afb6f702b33a0d2a P 8a9f21049c9c4db48d48db896dcce03d: Time spent starting tablet: real 0.016s user 0.015s sys 0.000s
I20260807 23:01:04.222543 21244 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.1:45459
I20260807 23:01:04.222579 21539 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.1:45459 every 8 connection(s)
I20260807 23:01:04.224007 21244 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-0/data/info.pb
I20260807 23:01:04.229306 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 21244
I20260807 23:01:04.229365 21544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:04.227083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:40547 (local address 127.25.94.1:53579)
0807 23:01:04.228020 (+ 937us) negotiation.cc:107] Waiting for socket to connect
0807 23:01:04.228282 (+ 262us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.62:40547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":674,"thread_start_us":399,"threads_started":1}
I20260807 23:01:04.230023 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.2:36939
--local_ip_for_outbound_sockets=127.25.94.2
--tserver_master_addrs=127.25.94.62:40547
--webserver_port=37971
--webserver_interface=127.25.94.2
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.2.keytab
--principal=kudu/127.25.94.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.20:35499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 23:01:04.230129 21540 heartbeater.cc:644] Failed to heartbeat to 127.25.94.62:40547 (0 consecutive failures): Network error: Failed to ping master at 127.25.94.62:40547: Client connection negotiation failed: client connection to 127.25.94.62:40547: connect: Connection refused (error 111)
I20260807 23:01:04.231070 21544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:04.230701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:40547 (local address 127.25.94.1:35361)
0807 23:01:04.230874 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0807 23:01:04.230959 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.62:40547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260807 23:01:04.232087 21544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:04.231712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:40547 (local address 127.25.94.1:54167)
0807 23:01:04.231881 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0807 23:01:04.231960 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.62:40547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260807 23:01:04.232435 21540 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260807 23:01:04.589812 21545 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 23:01:04.590129 21545 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 23:01:04.590229 21545 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 23:01:04.597978 21545 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 23:01:04.598106 21545 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 23:01:04.598348 21545 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.2
I20260807 23:01:04.608321 21545 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:35499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.2.keytab
--principal=kudu/127.25.94.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.2:36939
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.94.2
--webserver_port=37971
--tserver_master_addrs=127.25.94.62:40547
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.2
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 23:01:04.609670 21545 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 23:01:04.610962 21545 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 23:01:04.624668 21559 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 23:01:04.624779 21556 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 23:01:04.624806 21554 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 23:01:04.625617 21545 server_base.cc:1034] running on GCE node
I20260807 23:01:04.626435 21545 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 23:01:04.627445 21545 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 23:01:04.628684 21545 hybrid_clock.cc:648] HybridClock initialized: now 1786143664628617 us; error 57 us; skew 500 ppm
I20260807 23:01:04.629012 21545 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 23:01:04 dist-test-slave-z8x9 krb5kdc[19243](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143664, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 23:01:04.632690 21545 init.cc:377] Logged in from keytab as kudu/127.25.94.2@KRBTEST.COM (short username kudu)
I20260807 23:01:04.634593 21545 webserver.cc:460] Webserver started at http://127.25.94.2:37971/ using document root <none> and password file <none>
I20260807 23:01:04.635363 21545 fs_manager.cc:359] Metadata directory not provided
I20260807 23:01:04.635485 21545 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 23:01:04.641439 21545 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260807 23:01:04.644832 21567 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 23:01:04.647177 21545 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260807 23:01:04.647399 21545 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-1/wal
uuid: "490f1fd977fb4fd586aed08107d9f266"
format_stamp: "Formatted at 2026-08-07 23:00:18 on dist-test-slave-z8x9"
server_key: "ad85a15d8b332a3dd74c4377ce274f8a"
server_key_iv: "6ea30c2da1c71d3dd026ef2728b7c3bb"
server_key_version: "encryptionkey@0"
I20260807 23:01:04.648429 21545 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 23:01:04.673736 21545 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 23:01:04.678519 21545 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 23:01:04.678845 21545 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 23:01:04.679917 21545 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 23:01:04.681404 21545 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 23:01:04.681535 21545 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 23:01:04.681663 21545 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 23:01:04.681769 21545 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 23:01:04.718405 21545 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.2:36939
I20260807 23:01:04.718499 21739 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.2:36939 every 8 connection(s)
I20260807 23:01:04.719858 21545 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-1/data/info.pb
I20260807 23:01:04.724754 21744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:04.722860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:40547 (local address 127.25.94.2:50731)
0807 23:01:04.723633 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0807 23:01:04.723873 (+ 240us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.62:40547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":313,"threads_started":1}
W20260807 23:01:04.725636 21742 heartbeater.cc:644] Failed to heartbeat to 127.25.94.62:40547 (0 consecutive failures): Network error: Failed to ping master at 127.25.94.62:40547: Client connection negotiation failed: client connection to 127.25.94.62:40547: connect: Connection refused (error 111)
I20260807 23:01:04.726595 21744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:04.726247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:40547 (local address 127.25.94.2:54863)
0807 23:01:04.726409 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0807 23:01:04.726483 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.62:40547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260807 23:01:04.727491 21744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:04.727127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:40547 (local address 127.25.94.2:57461)
0807 23:01:04.727304 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0807 23:01:04.727382 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.62:40547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
W20260807 23:01:04.727746 21742 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260807 23:01:04.728148 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 21545
I20260807 23:01:04.728771 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.3:41199
--local_ip_for_outbound_sockets=127.25.94.3
--tserver_master_addrs=127.25.94.62:40547
--webserver_port=33175
--webserver_interface=127.25.94.3
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.3.keytab
--principal=kudu/127.25.94.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.20:35499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 23:01:05.074605 21746 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 23:01:05.074942 21746 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 23:01:05.075102 21746 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 23:01:05.082674 21746 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 23:01:05.082808 21746 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 23:01:05.082988 21746 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.3
I20260807 23:01:05.094045 21746 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:35499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.3.keytab
--principal=kudu/127.25.94.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.3:41199
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.94.3
--webserver_port=33175
--tserver_master_addrs=127.25.94.62:40547
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.3
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 23:01:05.095477 21746 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 23:01:05.096935 21746 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 23:01:05.112506 21760 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 23:01:05.112510 21761 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 23:01:05.112777 21768 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 23:01:05.113296 21746 server_base.cc:1034] running on GCE node
I20260807 23:01:05.114183 21746 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 23:01:05.115906 21746 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 23:01:05.117326 21746 hybrid_clock.cc:648] HybridClock initialized: now 1786143665117174 us; error 94 us; skew 500 ppm
I20260807 23:01:05.117707 21746 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 23:01:05 dist-test-slave-z8x9 krb5kdc[19243](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143665, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 23:01:05.121315 21746 init.cc:377] Logged in from keytab as kudu/127.25.94.3@KRBTEST.COM (short username kudu)
I20260807 23:01:05.123224 21746 webserver.cc:460] Webserver started at http://127.25.94.3:33175/ using document root <none> and password file <none>
I20260807 23:01:05.124058 21746 fs_manager.cc:359] Metadata directory not provided
I20260807 23:01:05.124228 21746 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 23:01:05.130878 21746 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260807 23:01:05.134087 21780 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 23:01:05.136057 21746 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260807 23:01:05.136389 21746 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-2/wal
uuid: "a49447d7b0024e12b538ea065f867cb5"
format_stamp: "Formatted at 2026-08-07 23:00:19 on dist-test-slave-z8x9"
server_key: "104d48c30a3e317d64e2257306c7c74f"
server_key_iv: "a256b9d5ac16994f5bb2160b184176ec"
server_key_version: "encryptionkey@0"
I20260807 23:01:05.137609 21746 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 23:01:05.157088 21746 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 23:01:05.163589 21746 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 23:01:05.163976 21746 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 23:01:05.165127 21746 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 23:01:05.166795 21746 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 23:01:05.166976 21746 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 23:01:05.167124 21746 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 23:01:05.167223 21746 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 23:01:05.205008 21746 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.3:41199
I20260807 23:01:05.205075 21967 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.3:41199 every 8 connection(s)
I20260807 23:01:05.206635 21746 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786143361779964-25976-0/minicluster-data/ts-2/data/info.pb
I20260807 23:01:05.210762 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 21746
I20260807 23:01:05.212572 21973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:05.210337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:40547 (local address 127.25.94.3:35703)
0807 23:01:05.211251 (+ 914us) negotiation.cc:107] Waiting for socket to connect
0807 23:01:05.211543 (+ 292us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.62:40547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":590,"thread_start_us":383,"threads_started":1}
W20260807 23:01:05.213672 21971 heartbeater.cc:644] Failed to heartbeat to 127.25.94.62:40547 (0 consecutive failures): Network error: Failed to ping master at 127.25.94.62:40547: Client connection negotiation failed: client connection to 127.25.94.62:40547: connect: Connection refused (error 111)
I20260807 23:01:05.214603 21973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:05.214207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:40547 (local address 127.25.94.3:34111)
0807 23:01:05.214419 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0807 23:01:05.214491 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.62:40547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260807 23:01:05.215688 21973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:05.215258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:40547 (local address 127.25.94.3:48921)
0807 23:01:05.215431 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0807 23:01:05.215547 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.62:40547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260807 23:01:05.216060 21971 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260807 23:01:05.234047 21976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:05.233068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:40547 (local address 127.25.94.1:43973)
0807 23:01:05.233675 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0807 23:01:05.233818 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.62:40547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":341,"threads_started":1}
I20260807 23:01:05.441227 21978 raft_consensus.cc:482] T 6930c65beb564053afb6f702b33a0d2a P 8a9f21049c9c4db48d48db896dcce03d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 23:01:05.441607 21978 raft_consensus.cc:504] T 6930c65beb564053afb6f702b33a0d2a P 8a9f21049c9c4db48d48db896dcce03d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a49447d7b0024e12b538ea065f867cb5" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 41199 } } peers { permanent_uuid: "490f1fd977fb4fd586aed08107d9f266" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 36939 } } peers { permanent_uuid: "8a9f21049c9c4db48d48db896dcce03d" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 45459 } }
I20260807 23:01:05.443635 21978 leader_election.cc:288] T 6930c65beb564053afb6f702b33a0d2a P 8a9f21049c9c4db48d48db896dcce03d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a49447d7b0024e12b538ea065f867cb5 (127.25.94.3:41199), 490f1fd977fb4fd586aed08107d9f266 (127.25.94.2:36939)
Aug 07 23:01:05 dist-test-slave-z8x9 krb5kdc[19243](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143664, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.1@KRBTEST.COM for kudu/127.25.94.2@KRBTEST.COM
Aug 07 23:01:05 dist-test-slave-z8x9 krb5kdc[19243](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143664, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.1@KRBTEST.COM for kudu/127.25.94.3@KRBTEST.COM
I20260807 23:01:05.473150 21976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:05.443897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.2:36939 (local address 127.25.94.1:42861)
0807 23:01:05.444244 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0807 23:01:05.444318 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0807 23:01:05.445046 (+ 728us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 23:01:05.449557 (+ 4511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 23:01:05.449568 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 23:01:05.450591 (+ 1023us) client_negotiation.cc:189] Negotiated authn=SASL
0807 23:01:05.451714 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:01:05.451743 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:01:05.454248 (+ 2505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:01:05.454253 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:01:05.456580 (+ 2327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:01:05.456598 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:01:05.458167 (+ 1569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:01:05.458173 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:01:05.458516 (+ 343us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:05.459767 (+ 1251us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 23:01:05.459812 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0807 23:01:05.463229 (+ 3417us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 23:01:05.468250 (+ 5021us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 23:01:05.468262 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 23:01:05.468281 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0807 23:01:05.469247 (+ 966us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 23:01:05.469824 (+ 577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 23:01:05.469829 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 23:01:05.469833 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 23:01:05.470147 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 23:01:05.471233 (+ 1086us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 23:01:05.471237 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 23:01:05.471721 (+ 484us) client_negotiation.cc:769] Sending connection context
0807 23:01:05.472079 (+ 358us) client_negotiation.cc:240] Negotiation successful
0807 23:01:05.472676 (+ 597us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"mutex_wait_us":342,"spinlock_wait_cycles":112000}
I20260807 23:01:05.473335 21981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:05.445006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.1:42861 (local address 127.25.94.2:36939)
0807 23:01:05.446339 (+ 1333us) server_negotiation.cc:206] Beginning negotiation
0807 23:01:05.446350 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 23:01:05.446384 (+ 34us) server_negotiation.cc:407] Connection header received
0807 23:01:05.448499 (+ 2115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:01:05.448512 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:01:05.449035 (+ 523us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:01:05.449463 (+ 428us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:01:05.451959 (+ 2496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:05.453975 (+ 2016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:05.456773 (+ 2798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:05.457888 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:05.458087 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:05.463614 (+ 5527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:01:05.463623 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:01:05.463663 (+ 40us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 23:01:05.463729 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:01:05.468023 (+ 4294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:01:05.469419 (+ 1396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:01:05.469429 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:01:05.469438 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:01:05.469677 (+ 239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:01:05.470310 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:01:05.470316 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:01:05.470318 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:01:05.471035 (+ 717us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:01:05.471223 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0807 23:01:05.472385 (+ 1162us) server_negotiation.cc:299] Negotiation successful
0807 23:01:05.472850 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1157,"thread_start_us":1050,"threads_started":1}
I20260807 23:01:05.473368 21980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:05.444231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.1:48873 (local address 127.25.94.3:41199)
0807 23:01:05.446150 (+ 1919us) server_negotiation.cc:206] Beginning negotiation
0807 23:01:05.446173 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0807 23:01:05.446818 (+ 645us) server_negotiation.cc:407] Connection header received
0807 23:01:05.448890 (+ 2072us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:01:05.448899 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:01:05.449309 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:01:05.449636 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:01:05.451959 (+ 2323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:05.453972 (+ 2013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:05.456773 (+ 2801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:05.457888 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:05.458087 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:05.463865 (+ 5778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:01:05.463877 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:01:05.463893 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 23:01:05.463951 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:01:05.468023 (+ 4072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:01:05.469419 (+ 1396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:01:05.469429 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:01:05.469438 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:01:05.469677 (+ 239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:01:05.470292 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:01:05.470303 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:01:05.470306 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:01:05.471035 (+ 729us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:01:05.471223 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0807 23:01:05.472385 (+ 1162us) server_negotiation.cc:299] Negotiation successful
0807 23:01:05.472850 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1650,"thread_start_us":1433,"threads_started":1}
I20260807 23:01:05.473238 21979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:05.443923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.3:41199 (local address 127.25.94.1:48873)
0807 23:01:05.446689 (+ 2766us) negotiation.cc:107] Waiting for socket to connect
0807 23:01:05.446704 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 23:01:05.446817 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 23:01:05.449600 (+ 2783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 23:01:05.449604 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 23:01:05.450591 (+ 987us) client_negotiation.cc:189] Negotiated authn=SASL
0807 23:01:05.451713 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:01:05.451743 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:01:05.454246 (+ 2503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:01:05.454282 (+ 36us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:01:05.456580 (+ 2298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:01:05.456598 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:01:05.458155 (+ 1557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:01:05.458163 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:01:05.458521 (+ 358us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:05.459767 (+ 1246us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 23:01:05.459809 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0807 23:01:05.463564 (+ 3755us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 23:01:05.468248 (+ 4684us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 23:01:05.468262 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 23:01:05.468281 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0807 23:01:05.469247 (+ 966us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 23:01:05.469823 (+ 576us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 23:01:05.469829 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 23:01:05.469833 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 23:01:05.470147 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 23:01:05.471201 (+ 1054us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 23:01:05.471210 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 23:01:05.471728 (+ 518us) client_negotiation.cc:769] Sending connection context
0807 23:01:05.472079 (+ 351us) client_negotiation.cc:240] Negotiation successful
0807 23:01:05.472823 (+ 744us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2625,"mutex_wait_us":604,"spinlock_wait_cycles":81280,"thread_start_us":2404,"threads_started":1}
W20260807 23:01:05.477519 21346 leader_election.cc:341] T 6930c65beb564053afb6f702b33a0d2a P 8a9f21049c9c4db48d48db896dcce03d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 490f1fd977fb4fd586aed08107d9f266 (127.25.94.2:36939): Not found: Tablet not found: 6930c65beb564053afb6f702b33a0d2a
W20260807 23:01:05.477896 21343 leader_election.cc:341] T 6930c65beb564053afb6f702b33a0d2a P 8a9f21049c9c4db48d48db896dcce03d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a49447d7b0024e12b538ea065f867cb5 (127.25.94.3:41199): Not found: Tablet not found: 6930c65beb564053afb6f702b33a0d2a
I20260807 23:01:05.478050 21343 leader_election.cc:302] T 6930c65beb564053afb6f702b33a0d2a P 8a9f21049c9c4db48d48db896dcce03d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a9f21049c9c4db48d48db896dcce03d; no voters: 490f1fd977fb4fd586aed08107d9f266, a49447d7b0024e12b538ea065f867cb5
I20260807 23:01:05.478567 21978 raft_consensus.cc:2726] T 6930c65beb564053afb6f702b33a0d2a P 8a9f21049c9c4db48d48db896dcce03d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 23:01:05.729334 21985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:05.728335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:40547 (local address 127.25.94.2:42001)
0807 23:01:05.728903 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0807 23:01:05.729053 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.62:40547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":310,"threads_started":1}
I20260807 23:01:06.217865 21986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:06.216895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:40547 (local address 127.25.94.3:45643)
0807 23:01:06.217522 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0807 23:01:06.217643 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.62:40547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":378,"threads_started":1}
I20260807 23:01:06.235921 21989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:06.235154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:40547 (local address 127.25.94.1:41723)
0807 23:01:06.235653 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0807 23:01:06.235744 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.62:40547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":264,"threads_started":1}
I20260807 23:01:06.731014 21990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:06.730224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:40547 (local address 127.25.94.2:36681)
0807 23:01:06.730776 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0807 23:01:06.730856 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.62:40547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":343,"threads_started":1}
I20260807 23:01:07.079236 21991 raft_consensus.cc:482] T 6930c65beb564053afb6f702b33a0d2a P 8a9f21049c9c4db48d48db896dcce03d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 23:01:07.079469 21991 raft_consensus.cc:504] T 6930c65beb564053afb6f702b33a0d2a P 8a9f21049c9c4db48d48db896dcce03d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a49447d7b0024e12b538ea065f867cb5" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 41199 } } peers { permanent_uuid: "490f1fd977fb4fd586aed08107d9f266" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 36939 } } peers { permanent_uuid: "8a9f21049c9c4db48d48db896dcce03d" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 45459 } }
I20260807 23:01:07.080248 21991 leader_election.cc:288] T 6930c65beb564053afb6f702b33a0d2a P 8a9f21049c9c4db48d48db896dcce03d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a49447d7b0024e12b538ea065f867cb5 (127.25.94.3:41199), 490f1fd977fb4fd586aed08107d9f266 (127.25.94.2:36939)
W20260807 23:01:07.081363 21343 leader_election.cc:341] T 6930c65beb564053afb6f702b33a0d2a P 8a9f21049c9c4db48d48db896dcce03d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a49447d7b0024e12b538ea065f867cb5 (127.25.94.3:41199): Not found: Tablet not found: 6930c65beb564053afb6f702b33a0d2a
W20260807 23:01:07.081526 21346 leader_election.cc:341] T 6930c65beb564053afb6f702b33a0d2a P 8a9f21049c9c4db48d48db896dcce03d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 490f1fd977fb4fd586aed08107d9f266 (127.25.94.2:36939): Not found: Tablet not found: 6930c65beb564053afb6f702b33a0d2a
I20260807 23:01:07.081664 21346 leader_election.cc:302] T 6930c65beb564053afb6f702b33a0d2a P 8a9f21049c9c4db48d48db896dcce03d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a9f21049c9c4db48d48db896dcce03d; no voters: 490f1fd977fb4fd586aed08107d9f266, a49447d7b0024e12b538ea065f867cb5
I20260807 23:01:07.081948 21991 raft_consensus.cc:2726] T 6930c65beb564053afb6f702b33a0d2a P 8a9f21049c9c4db48d48db896dcce03d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 23:01:07.219794 21992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:07.218964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:40547 (local address 127.25.94.3:42463)
0807 23:01:07.219494 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0807 23:01:07.219584 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.62:40547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":277,"threads_started":1}
I20260807 23:01:07.237968 21993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:07.236977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:40547 (local address 127.25.94.1:40981)
0807 23:01:07.237614 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0807 23:01:07.237725 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.62:40547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":366,"threads_started":1}
I20260807 23:01:07.732774 21994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:07.731963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:40547 (local address 127.25.94.2:41157)
0807 23:01:07.732480 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0807 23:01:07.732571 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.62:40547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":284,"threads_started":1}
I20260807 23:01:08.222198 21997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:08.221223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:40547 (local address 127.25.94.3:42795)
0807 23:01:08.221845 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0807 23:01:08.221955 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.62:40547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":363,"threads_started":1}
I20260807 23:01:08.240320 21998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:08.239342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:40547 (local address 127.25.94.1:54755)
0807 23:01:08.240033 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0807 23:01:08.240142 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.62:40547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":513,"threads_started":1}
I20260807 23:01:08.734727 22002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:08.733897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:40547 (local address 127.25.94.2:46071)
0807 23:01:08.734415 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0807 23:01:08.734522 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.62:40547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":254,"threads_started":1}
I20260807 23:01:09.224261 22007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:09.223290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:40547 (local address 127.25.94.3:54503)
0807 23:01:09.223949 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0807 23:01:09.224052 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.62:40547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":392,"threads_started":1}
I20260807 23:01:09.242643 22008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:09.241709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:40547 (local address 127.25.94.1:48301)
0807 23:01:09.242267 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0807 23:01:09.242359 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.62:40547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":256,"threads_started":1}
I20260807 23:01:09.736610 22009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:09.735678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:40547 (local address 127.25.94.2:46625)
0807 23:01:09.736355 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0807 23:01:09.736440 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.62:40547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":264,"threads_started":1}
I20260807 23:01:09.933418 22011 raft_consensus.cc:482] T 6930c65beb564053afb6f702b33a0d2a P 8a9f21049c9c4db48d48db896dcce03d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 23:01:09.933645 22011 raft_consensus.cc:504] T 6930c65beb564053afb6f702b33a0d2a P 8a9f21049c9c4db48d48db896dcce03d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a49447d7b0024e12b538ea065f867cb5" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 41199 } } peers { permanent_uuid: "490f1fd977fb4fd586aed08107d9f266" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 36939 } } peers { permanent_uuid: "8a9f21049c9c4db48d48db896dcce03d" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 45459 } }
I20260807 23:01:09.934265 22011 leader_election.cc:288] T 6930c65beb564053afb6f702b33a0d2a P 8a9f21049c9c4db48d48db896dcce03d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a49447d7b0024e12b538ea065f867cb5 (127.25.94.3:41199), 490f1fd977fb4fd586aed08107d9f266 (127.25.94.2:36939)
W20260807 23:01:09.935361 21346 leader_election.cc:341] T 6930c65beb564053afb6f702b33a0d2a P 8a9f21049c9c4db48d48db896dcce03d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 490f1fd977fb4fd586aed08107d9f266 (127.25.94.2:36939): Not found: Tablet not found: 6930c65beb564053afb6f702b33a0d2a
W20260807 23:01:09.935559 21343 leader_election.cc:341] T 6930c65beb564053afb6f702b33a0d2a P 8a9f21049c9c4db48d48db896dcce03d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a49447d7b0024e12b538ea065f867cb5 (127.25.94.3:41199): Not found: Tablet not found: 6930c65beb564053afb6f702b33a0d2a
I20260807 23:01:09.935680 21343 leader_election.cc:302] T 6930c65beb564053afb6f702b33a0d2a P 8a9f21049c9c4db48d48db896dcce03d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a9f21049c9c4db48d48db896dcce03d; no voters: 490f1fd977fb4fd586aed08107d9f266, a49447d7b0024e12b538ea065f867cb5
I20260807 23:01:09.935894 22011 raft_consensus.cc:2726] T 6930c65beb564053afb6f702b33a0d2a P 8a9f21049c9c4db48d48db896dcce03d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 23:01:10.226263 22014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:10.225460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:40547 (local address 127.25.94.3:49597)
0807 23:01:10.225975 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0807 23:01:10.226074 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.62:40547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":289,"threads_started":1}
I20260807 23:01:10.244570 22015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:10.243626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:40547 (local address 127.25.94.1:39683)
0807 23:01:10.244256 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0807 23:01:10.244362 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.62:40547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":387,"threads_started":1}
I20260807 23:01:10.738610 22016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:10.737676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:40547 (local address 127.25.94.2:34731)
0807 23:01:10.738346 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0807 23:01:10.738428 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.62:40547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":337,"threads_started":1}
I20260807 23:01:11.228462 22017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:11.227407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:40547 (local address 127.25.94.3:39459)
0807 23:01:11.228119 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0807 23:01:11.228253 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.62:40547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":417,"threads_started":1}
I20260807 23:01:11.246527 22018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:11.245668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:40547 (local address 127.25.94.1:41255)
0807 23:01:11.246237 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0807 23:01:11.246326 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.62:40547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":330,"threads_started":1}
I20260807 23:01:11.740388 22019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:11.739574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:40547 (local address 127.25.94.2:37009)
0807 23:01:11.740086 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0807 23:01:11.740181 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.62:40547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":221,"threads_started":1}
I20260807 23:01:12.230813 22020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:12.229769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:40547 (local address 127.25.94.3:41549)
0807 23:01:12.230480 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0807 23:01:12.230580 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.62:40547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":334,"threads_started":1}
I20260807 23:01:12.248368 22026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:12.247583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:40547 (local address 127.25.94.1:45819)
0807 23:01:12.248089 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0807 23:01:12.248176 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.62:40547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":241,"threads_started":1}
I20260807 23:01:12.742261 22028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:12.741428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:40547 (local address 127.25.94.2:41311)
0807 23:01:12.741968 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0807 23:01:12.742066 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.62:40547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":250,"threads_started":1}
I20260807 23:01:13.232860 22030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:13.232039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:40547 (local address 127.25.94.3:39223)
0807 23:01:13.232561 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0807 23:01:13.232654 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.62:40547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":271,"threads_started":1}
I20260807 23:01:13.250330 22033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:13.249557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:40547 (local address 127.25.94.1:36869)
0807 23:01:13.250044 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0807 23:01:13.250143 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.62:40547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":269,"threads_started":1}
I20260807 23:01:13.548581 22034 raft_consensus.cc:482] T 6930c65beb564053afb6f702b33a0d2a P 8a9f21049c9c4db48d48db896dcce03d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 23:01:13.548900 22034 raft_consensus.cc:504] T 6930c65beb564053afb6f702b33a0d2a P 8a9f21049c9c4db48d48db896dcce03d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a49447d7b0024e12b538ea065f867cb5" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 41199 } } peers { permanent_uuid: "490f1fd977fb4fd586aed08107d9f266" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 36939 } } peers { permanent_uuid: "8a9f21049c9c4db48d48db896dcce03d" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 45459 } }
I20260807 23:01:13.549610 22034 leader_election.cc:288] T 6930c65beb564053afb6f702b33a0d2a P 8a9f21049c9c4db48d48db896dcce03d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a49447d7b0024e12b538ea065f867cb5 (127.25.94.3:41199), 490f1fd977fb4fd586aed08107d9f266 (127.25.94.2:36939)
W20260807 23:01:13.550774 21346 leader_election.cc:341] T 6930c65beb564053afb6f702b33a0d2a P 8a9f21049c9c4db48d48db896dcce03d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 490f1fd977fb4fd586aed08107d9f266 (127.25.94.2:36939): Not found: Tablet not found: 6930c65beb564053afb6f702b33a0d2a
W20260807 23:01:13.550976 21343 leader_election.cc:341] T 6930c65beb564053afb6f702b33a0d2a P 8a9f21049c9c4db48d48db896dcce03d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a49447d7b0024e12b538ea065f867cb5 (127.25.94.3:41199): Not found: Tablet not found: 6930c65beb564053afb6f702b33a0d2a
I20260807 23:01:13.551096 21343 leader_election.cc:302] T 6930c65beb564053afb6f702b33a0d2a P 8a9f21049c9c4db48d48db896dcce03d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a9f21049c9c4db48d48db896dcce03d; no voters: 490f1fd977fb4fd586aed08107d9f266, a49447d7b0024e12b538ea065f867cb5
I20260807 23:01:13.551338 22034 raft_consensus.cc:2726] T 6930c65beb564053afb6f702b33a0d2a P 8a9f21049c9c4db48d48db896dcce03d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 23:01:13.744697 22035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:13.743928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:40547 (local address 127.25.94.2:35331)
0807 23:01:13.744431 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0807 23:01:13.744532 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.62:40547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":259,"threads_started":1}
I20260807 23:01:14.235138 22037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:14.234201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:40547 (local address 127.25.94.3:38727)
0807 23:01:14.234830 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0807 23:01:14.234941 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.62:40547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":310,"threads_started":1}
I20260807 23:01:14.252451 22038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:14.251563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:40547 (local address 127.25.94.1:42197)
0807 23:01:14.252180 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0807 23:01:14.252285 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.62:40547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":368,"threads_started":1}
I20260807 23:01:14.746438 22039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:14.745636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:40547 (local address 127.25.94.2:58517)
0807 23:01:14.746124 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0807 23:01:14.746211 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.62:40547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":255,"threads_started":1}
I20260807 23:01:15.237131 22040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:15.236302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:40547 (local address 127.25.94.3:36265)
0807 23:01:15.236856 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0807 23:01:15.236946 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.62:40547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":292,"threads_started":1}
I20260807 23:01:15.254514 22041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:15.253674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:40547 (local address 127.25.94.1:51781)
0807 23:01:15.254216 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0807 23:01:15.254330 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.62:40547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":301,"threads_started":1}
I20260807 23:01:15.292284 22042 raft_consensus.cc:482] T 6930c65beb564053afb6f702b33a0d2a P 8a9f21049c9c4db48d48db896dcce03d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 23:01:15.292502 22042 raft_consensus.cc:504] T 6930c65beb564053afb6f702b33a0d2a P 8a9f21049c9c4db48d48db896dcce03d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a49447d7b0024e12b538ea065f867cb5" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 41199 } } peers { permanent_uuid: "490f1fd977fb4fd586aed08107d9f266" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 36939 } } peers { permanent_uuid: "8a9f21049c9c4db48d48db896dcce03d" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 45459 } }
I20260807 23:01:15.293246 22042 leader_election.cc:288] T 6930c65beb564053afb6f702b33a0d2a P 8a9f21049c9c4db48d48db896dcce03d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a49447d7b0024e12b538ea065f867cb5 (127.25.94.3:41199), 490f1fd977fb4fd586aed08107d9f266 (127.25.94.2:36939)
W20260807 23:01:15.294353 21343 leader_election.cc:341] T 6930c65beb564053afb6f702b33a0d2a P 8a9f21049c9c4db48d48db896dcce03d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a49447d7b0024e12b538ea065f867cb5 (127.25.94.3:41199): Not found: Tablet not found: 6930c65beb564053afb6f702b33a0d2a
W20260807 23:01:15.294561 21346 leader_election.cc:341] T 6930c65beb564053afb6f702b33a0d2a P 8a9f21049c9c4db48d48db896dcce03d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 490f1fd977fb4fd586aed08107d9f266 (127.25.94.2:36939): Not found: Tablet not found: 6930c65beb564053afb6f702b33a0d2a
I20260807 23:01:15.294725 21346 leader_election.cc:302] T 6930c65beb564053afb6f702b33a0d2a P 8a9f21049c9c4db48d48db896dcce03d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a9f21049c9c4db48d48db896dcce03d; no voters: 490f1fd977fb4fd586aed08107d9f266, a49447d7b0024e12b538ea065f867cb5
I20260807 23:01:15.295024 22042 raft_consensus.cc:2726] T 6930c65beb564053afb6f702b33a0d2a P 8a9f21049c9c4db48d48db896dcce03d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 23:01:15.748222 22043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:15.747510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:40547 (local address 127.25.94.2:52941)
0807 23:01:15.747968 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0807 23:01:15.748050 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.62:40547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":235,"threads_started":1}
I20260807 23:01:16.238996 22045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:16.238171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:40547 (local address 127.25.94.3:37807)
0807 23:01:16.238728 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0807 23:01:16.238822 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.62:40547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":279,"threads_started":1}
I20260807 23:01:16.256526 22046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:16.255675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:40547 (local address 127.25.94.1:38719)
0807 23:01:16.256213 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0807 23:01:16.256334 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.62:40547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":303,"threads_started":1}
I20260807 23:01:16.750082 22048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:16.749326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:40547 (local address 127.25.94.2:35811)
0807 23:01:16.749784 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0807 23:01:16.749882 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.62:40547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":232,"threads_started":1}
I20260807 23:01:17.240787 22051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:17.240040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:40547 (local address 127.25.94.3:58559)
0807 23:01:17.240527 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0807 23:01:17.240610 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.62:40547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":270,"threads_started":1}
I20260807 23:01:17.258512 22055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:17.257735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:40547 (local address 127.25.94.1:58341)
0807 23:01:17.258236 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0807 23:01:17.258323 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.62:40547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":275,"threads_started":1}
I20260807 23:01:17.751884 22056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:17.751107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:40547 (local address 127.25.94.2:53733)
0807 23:01:17.751601 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0807 23:01:17.751704 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.62:40547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":251,"threads_started":1}
I20260807 23:01:18.242905 22059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:18.241781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:40547 (local address 127.25.94.3:39259)
0807 23:01:18.242504 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0807 23:01:18.242645 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.62:40547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":409,"threads_started":1}
I20260807 23:01:18.260648 22061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:18.259581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:40547 (local address 127.25.94.1:39927)
0807 23:01:18.260258 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0807 23:01:18.260390 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.62:40547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":315,"threads_started":1}
I20260807 23:01:18.753849 22063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:18.752937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:40547 (local address 127.25.94.2:49361)
0807 23:01:18.753551 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0807 23:01:18.753636 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.62:40547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":368,"threads_started":1}
I20260807 23:01:19.244912 22064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:19.243892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:40547 (local address 127.25.94.3:39889)
0807 23:01:19.244584 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0807 23:01:19.244690 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.62:40547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":240,"threads_started":1}
I20260807 23:01:19.262844 22066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:19.262017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:40547 (local address 127.25.94.1:60701)
0807 23:01:19.262574 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0807 23:01:19.262665 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.62:40547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":282,"threads_started":1}
I20260807 23:01:19.755873 22068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:19.755061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:40547 (local address 127.25.94.2:44761)
0807 23:01:19.755571 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0807 23:01:19.755695 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.62:40547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":264,"threads_started":1}
I20260807 23:01:20.223435 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 21244
I20260807 23:01:20.243018 22069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:20.224843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41450 (local address 127.25.94.1:45459)
0807 23:01:20.226895 (+ 2052us) server_negotiation.cc:206] Beginning negotiation
0807 23:01:20.226905 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 23:01:20.226939 (+ 34us) server_negotiation.cc:407] Connection header received
0807 23:01:20.227010 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:01:20.227019 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:01:20.227326 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:01:20.227513 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:01:20.229351 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:20.230656 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:20.232050 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:20.232633 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:20.232794 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:20.235915 (+ 3121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:01:20.235931 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:01:20.235948 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 23:01:20.235992 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:01:20.239574 (+ 3582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:01:20.240595 (+ 1021us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:01:20.240603 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:01:20.240605 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:01:20.240746 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:01:20.241221 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:01:20.241234 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:01:20.241237 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:01:20.241640 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:01:20.241974 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0807 23:01:20.242390 (+ 416us) server_negotiation.cc:299] Negotiation successful
0807 23:01:20.242661 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1871,"thread_start_us":1720,"threads_started":1}
I20260807 23:01:20.244539 21531 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 23:01:20.247568 22072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:20.246365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:40547 (local address 127.25.94.3:48915)
0807 23:01:20.247029 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0807 23:01:20.247154 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.62:40547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":418,"threads_started":1}
I20260807 23:01:20.341806 22073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:20.340596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:40547 (local address 127.25.94.1:48967)
0807 23:01:20.341311 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0807 23:01:20.341455 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.62:40547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":274,"threads_started":1}
I20260807 23:01:20.342162 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 21244
I20260807 23:01:20.370888 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 21545
I20260807 23:01:20.386233 22076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:20.371755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55252 (local address 127.25.94.2:36939)
0807 23:01:20.372335 (+ 580us) server_negotiation.cc:206] Beginning negotiation
0807 23:01:20.372340 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 23:01:20.372358 (+ 18us) server_negotiation.cc:407] Connection header received
0807 23:01:20.372416 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:01:20.372420 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:01:20.372482 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:01:20.372586 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:01:20.374323 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:20.375166 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:20.376435 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:20.376899 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:20.377002 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:20.380044 (+ 3042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:01:20.380051 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:01:20.380055 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 23:01:20.380097 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:01:20.383133 (+ 3036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:01:20.383859 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:01:20.383864 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:01:20.383867 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:01:20.384015 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:01:20.384373 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:01:20.384377 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:01:20.384380 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:01:20.384724 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:01:20.384941 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0807 23:01:20.385633 (+ 692us) server_negotiation.cc:299] Negotiation successful
0807 23:01:20.385878 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":490,"thread_start_us":336,"threads_started":1}
I20260807 23:01:20.387038 21733 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 23:01:20.473470 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 21545
I20260807 23:01:20.500450 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 21746
I20260807 23:01:20.515300 22078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:20.501311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46950 (local address 127.25.94.3:41199)
0807 23:01:20.501909 (+ 598us) server_negotiation.cc:206] Beginning negotiation
0807 23:01:20.501913 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 23:01:20.501930 (+ 17us) server_negotiation.cc:407] Connection header received
0807 23:01:20.501985 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:01:20.501988 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:01:20.502057 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:01:20.502162 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:01:20.503672 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:20.504524 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:20.505982 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:20.506429 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:20.506530 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:20.509623 (+ 3093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:01:20.509629 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:01:20.509633 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 23:01:20.509671 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:01:20.512601 (+ 2930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:01:20.513468 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:01:20.513472 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:01:20.513474 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:01:20.513646 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:01:20.513983 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:01:20.513986 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:01:20.513988 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:01:20.514245 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:01:20.514474 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0807 23:01:20.514798 (+ 324us) server_negotiation.cc:299] Negotiation successful
0807 23:01:20.514990 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":484,"thread_start_us":340,"threads_started":1}
I20260807 23:01:20.516251 21959 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 23:01:20.600934 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 21746
2026-08-07T23:01:20Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (65482 ms)
[----------] 30 tests from SecurityITest (318782 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786143361779964-25976-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 23:01:20 dist-test-slave-z8x9 krb5kdc[22086](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 23:01:20 dist-test-slave-z8x9 krb5kdc[22086](info): set up 2 sockets
Aug 07 23:01:20 dist-test-slave-z8x9 krb5kdc[22086](info): commencing operation
krb5kdc: starting...
W20260807 23:01:21.994987 25976 mini_kdc.cc:121] Time spent starting KDC: real 1.343s user 0.000s sys 0.042s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786143361779964-25976-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786143361779964-25976-0/krb5kdc/test-user.keytab.
Aug 07 23:01:22 dist-test-slave-z8x9 krb5kdc[22086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143682, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T23:01:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T23:01:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.94.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.62.keytab.
Entry for principal kudu/127.25.94.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.62.keytab.
I20260807 23:01:22.321046 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786143361779964-25976-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786143361779964-25976-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786143361779964-25976-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.62:39399
--webserver_interface=127.25.94.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.62.keytab
--principal=kudu/127.25.94.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.20:38015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.94.62:39399
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786143361779964-25976-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786143361779964-25976-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786143361779964-25976-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 23:01:22.681597 22105 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 23:01:22.681892 22105 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 23:01:22.681983 22105 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 23:01:22.689445 22105 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 23:01:22.689561 22105 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 23:01:22.689613 22105 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 23:01:22.689672 22105 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 23:01:22.689718 22105 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 23:01:22.699824 22105 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:38015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786143361779964-25976-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786143361779964-25976-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.94.62:39399
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.62.keytab
--principal=kudu/127.25.94.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.62:39399
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.94.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786143361779964-25976-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 23:01:22.701093 22105 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 23:01:22.702385 22105 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 23:01:22.715507 22114 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 23:01:22.715806 22119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 23:01:22.715853 22116 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 23:01:22.716115 22105 server_base.cc:1034] running on GCE node
I20260807 23:01:22.717257 22105 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 23:01:22.718912 22105 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 23:01:22.720168 22105 hybrid_clock.cc:648] HybridClock initialized: now 1786143682720104 us; error 42 us; skew 500 ppm
I20260807 23:01:22.720672 22105 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 23:01:22 dist-test-slave-z8x9 krb5kdc[22086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143682, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 23:01:22.724057 22105 init.cc:377] Logged in from keytab as kudu/127.25.94.62@KRBTEST.COM (short username kudu)
I20260807 23:01:22.725696 22105 webserver.cc:460] Webserver started at http://127.25.94.62:40125/ using document root <none> and password file <none>
I20260807 23:01:22.726408 22105 fs_manager.cc:359] Metadata directory not provided
I20260807 23:01:22.726562 22105 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 23:01:22.726876 22105 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 23:01:22.729990 22105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786143361779964-25976-0/minicluster-data/master-0/data/instance:
uuid: "ed2e823495994a278850048096d62cbc"
format_stamp: "Formatted at 2026-08-07 23:01:22 on dist-test-slave-z8x9"
server_key: "e07baf7609f32b3f229e55460459c2e8"
server_key_iv: "aefdede7782c5b2859120d2402501a4a"
server_key_version: "encryptionkey@0"
I20260807 23:01:22.730780 22105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786143361779964-25976-0/minicluster-data/master-0/wal/instance:
uuid: "ed2e823495994a278850048096d62cbc"
format_stamp: "Formatted at 2026-08-07 23:01:22 on dist-test-slave-z8x9"
server_key: "e07baf7609f32b3f229e55460459c2e8"
server_key_iv: "aefdede7782c5b2859120d2402501a4a"
server_key_version: "encryptionkey@0"
I20260807 23:01:22.736217 22105 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260807 23:01:22.739845 22131 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 23:01:22.741255 22105 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 23:01:22.741499 22105 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786143361779964-25976-0/minicluster-data/master-0/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786143361779964-25976-0/minicluster-data/master-0/wal
uuid: "ed2e823495994a278850048096d62cbc"
format_stamp: "Formatted at 2026-08-07 23:01:22 on dist-test-slave-z8x9"
server_key: "e07baf7609f32b3f229e55460459c2e8"
server_key_iv: "aefdede7782c5b2859120d2402501a4a"
server_key_version: "encryptionkey@0"
I20260807 23:01:22.741729 22105 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786143361779964-25976-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786143361779964-25976-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786143361779964-25976-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 23:01:22.774048 22105 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 23:01:22.778168 22105 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 23:01:22.778502 22105 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 23:01:22.797698 22105 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.62:39399
I20260807 23:01:22.797739 22216 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.62:39399 every 8 connection(s)
I20260807 23:01:22.799096 22105 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
I20260807 23:01:22.802716 22218 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 23:01:22.805025 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 22105
I20260807 23:01:22.805295 25976 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786143361779964-25976-0/minicluster-data/master-0/wal/instance
I20260807 23:01:22.805991 25976 external_mini_cluster.cc:1428] Setting key ca51855c23d9011508b47f6c2e73e8c2
I20260807 23:01:22.814987 22218 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed2e823495994a278850048096d62cbc: Bootstrap starting.
I20260807 23:01:22.818863 22218 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ed2e823495994a278850048096d62cbc: Neither blocks nor log segments found. Creating new log.
I20260807 23:01:22.820170 22218 log.cc:824] T 00000000000000000000000000000000 P ed2e823495994a278850048096d62cbc: Log is configured to *not* fsync() on all Append() calls
Aug 07 23:01:22 dist-test-slave-z8x9 krb5kdc[22086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143682, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.62@KRBTEST.COM
I20260807 23:01:22.823361 22218 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed2e823495994a278850048096d62cbc: No bootstrap required, opened a new log
I20260807 23:01:22.830856 22227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:22.808296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43324 (local address 127.25.94.62:39399)
0807 23:01:22.809091 (+ 795us) server_negotiation.cc:206] Beginning negotiation
0807 23:01:22.809103 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 23:01:22.809144 (+ 41us) server_negotiation.cc:407] Connection header received
0807 23:01:22.810872 (+ 1728us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:01:22.810894 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:01:22.811292 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:01:22.811596 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:01:22.813523 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:22.814810 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:22.816713 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:22.817474 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:22.817603 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:22.822959 (+ 5356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:01:22.822967 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:01:22.822980 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 23:01:22.823014 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:01:22.826457 (+ 3443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:01:22.827390 (+ 933us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:01:22.827396 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:01:22.827402 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:01:22.827569 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:01:22.828004 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:01:22.828008 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:01:22.828010 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:01:22.828590 (+ 580us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:01:22.828854 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0807 23:01:22.829407 (+ 553us) server_negotiation.cc:299] Negotiation successful
0807 23:01:22.829799 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590,"thread_start_us":389,"threads_started":1}
I20260807 23:01:22.832695 22218 raft_consensus.cc:348] T 00000000000000000000000000000000 P ed2e823495994a278850048096d62cbc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed2e823495994a278850048096d62cbc" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 39399 } }
I20260807 23:01:22.833082 22218 raft_consensus.cc:374] T 00000000000000000000000000000000 P ed2e823495994a278850048096d62cbc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 23:01:22.833221 22218 raft_consensus.cc:729] T 00000000000000000000000000000000 P ed2e823495994a278850048096d62cbc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed2e823495994a278850048096d62cbc, State: Initialized, Role: FOLLOWER
I20260807 23:01:22.833786 22218 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed2e823495994a278850048096d62cbc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed2e823495994a278850048096d62cbc" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 39399 } }
I20260807 23:01:22.834040 22218 raft_consensus.cc:388] T 00000000000000000000000000000000 P ed2e823495994a278850048096d62cbc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 23:01:22.834192 22218 raft_consensus.cc:482] T 00000000000000000000000000000000 P ed2e823495994a278850048096d62cbc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 23:01:22.834372 22218 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ed2e823495994a278850048096d62cbc [term 0 FOLLOWER]: Advancing to term 1
I20260807 23:01:22.836522 22218 raft_consensus.cc:504] T 00000000000000000000000000000000 P ed2e823495994a278850048096d62cbc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed2e823495994a278850048096d62cbc" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 39399 } }
I20260807 23:01:22.837018 22218 leader_election.cc:302] T 00000000000000000000000000000000 P ed2e823495994a278850048096d62cbc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ed2e823495994a278850048096d62cbc; no voters:
I20260807 23:01:22.837668 22218 leader_election.cc:288] T 00000000000000000000000000000000 P ed2e823495994a278850048096d62cbc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 23:01:22.837841 22229 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ed2e823495994a278850048096d62cbc [term 1 FOLLOWER]: Leader election won for term 1
I20260807 23:01:22.839015 22229 raft_consensus.cc:686] T 00000000000000000000000000000000 P ed2e823495994a278850048096d62cbc [term 1 LEADER]: Becoming Leader. State: Replica: ed2e823495994a278850048096d62cbc, State: Running, Role: LEADER
I20260807 23:01:22.839586 22229 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed2e823495994a278850048096d62cbc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed2e823495994a278850048096d62cbc" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 39399 } }
I20260807 23:01:22.839916 22218 sys_catalog.cc:564] T 00000000000000000000000000000000 P ed2e823495994a278850048096d62cbc [sys.catalog]: configured and running, proceeding with master startup.
I20260807 23:01:22.842417 22231 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed2e823495994a278850048096d62cbc [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed2e823495994a278850048096d62cbc. Latest consensus state: current_term: 1 leader_uuid: "ed2e823495994a278850048096d62cbc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed2e823495994a278850048096d62cbc" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 39399 } } }
I20260807 23:01:22.842835 22231 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed2e823495994a278850048096d62cbc [sys.catalog]: This master's current role is: LEADER
I20260807 23:01:22.842866 22230 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed2e823495994a278850048096d62cbc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ed2e823495994a278850048096d62cbc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed2e823495994a278850048096d62cbc" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 39399 } } }
I20260807 23:01:22.843231 22230 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed2e823495994a278850048096d62cbc [sys.catalog]: This master's current role is: LEADER
I20260807 23:01:22.844081 22244 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 23:01:22.850610 22244 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 23:01:22.860622 22244 catalog_manager.cc:1347] Generated new cluster ID: de1e90a3644343f49658f96f9d974853
I20260807 23:01:22.860767 22244 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 23:01:22.869446 22244 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 23:01:22.871379 22244 catalog_manager.cc:1504] Loading token signing keys...
I20260807 23:01:22.880903 22244 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ed2e823495994a278850048096d62cbc: Generated new TSK 0
I20260807 23:01:22.881785 22244 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.94.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.1.keytab.
Entry for principal kudu/127.25.94.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.1.keytab.
I20260807 23:01:22.972558 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786143361779964-25976-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786143361779964-25976-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786143361779964-25976-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786143361779964-25976-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.1:0
--local_ip_for_outbound_sockets=127.25.94.1
--webserver_interface=127.25.94.1
--webserver_port=0
--tserver_master_addrs=127.25.94.62:39399
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.1.keytab
--principal=kudu/127.25.94.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.20:38015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786143361779964-25976-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786143361779964-25976-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786143361779964-25976-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 23:01:23.347754 22259 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 23:01:23.348085 22259 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 23:01:23.348245 22259 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 23:01:23.355746 22259 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 23:01:23.355875 22259 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 23:01:23.356082 22259 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.1
I20260807 23:01:23.366079 22259 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:38015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786143361779964-25976-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786143361779964-25976-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.1.keytab
--principal=kudu/127.25.94.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786143361779964-25976-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.94.1
--webserver_port=0
--tserver_master_addrs=127.25.94.62:39399
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.1
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786143361779964-25976-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 23:01:23.367481 22259 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 23:01:23.368810 22259 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 23:01:23.383320 22268 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 23:01:23.383385 22270 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 23:01:23.383291 22274 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 23:01:23.384219 22259 server_base.cc:1034] running on GCE node
I20260807 23:01:23.384896 22259 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 23:01:23.386075 22259 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 23:01:23.387319 22259 hybrid_clock.cc:648] HybridClock initialized: now 1786143683387258 us; error 57 us; skew 500 ppm
I20260807 23:01:23.387660 22259 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 23:01:23 dist-test-slave-z8x9 krb5kdc[22086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143683, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 23:01:23.395058 22259 init.cc:377] Logged in from keytab as kudu/127.25.94.1@KRBTEST.COM (short username kudu)
I20260807 23:01:23.396668 22259 webserver.cc:460] Webserver started at http://127.25.94.1:34973/ using document root <none> and password file <none>
I20260807 23:01:23.397532 22259 fs_manager.cc:359] Metadata directory not provided
I20260807 23:01:23.397696 22259 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 23:01:23.398042 22259 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 23:01:23.401108 22259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786143361779964-25976-0/minicluster-data/ts-0/data/instance:
uuid: "49bb149190364633b51b66a547f21af1"
format_stamp: "Formatted at 2026-08-07 23:01:23 on dist-test-slave-z8x9"
server_key: "ababf6714b9b581a216cbac943dde576"
server_key_iv: "646d20d35d950e83be71a14c7352f0c7"
server_key_version: "encryptionkey@0"
I20260807 23:01:23.402216 22259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786143361779964-25976-0/minicluster-data/ts-0/wal/instance:
uuid: "49bb149190364633b51b66a547f21af1"
format_stamp: "Formatted at 2026-08-07 23:01:23 on dist-test-slave-z8x9"
server_key: "ababf6714b9b581a216cbac943dde576"
server_key_iv: "646d20d35d950e83be71a14c7352f0c7"
server_key_version: "encryptionkey@0"
I20260807 23:01:23.407593 22259 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260807 23:01:23.411054 22285 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 23:01:23.412446 22259 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 23:01:23.412649 22259 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786143361779964-25976-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786143361779964-25976-0/minicluster-data/ts-0/wal
uuid: "49bb149190364633b51b66a547f21af1"
format_stamp: "Formatted at 2026-08-07 23:01:23 on dist-test-slave-z8x9"
server_key: "ababf6714b9b581a216cbac943dde576"
server_key_iv: "646d20d35d950e83be71a14c7352f0c7"
server_key_version: "encryptionkey@0"
I20260807 23:01:23.412876 22259 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786143361779964-25976-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786143361779964-25976-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786143361779964-25976-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 23:01:23.433346 22259 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 23:01:23.437987 22259 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 23:01:23.438349 22259 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 23:01:23.439404 22259 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 23:01:23.440898 22259 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 23:01:23.441030 22259 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 23:01:23.441176 22259 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 23:01:23.441263 22259 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260807 23:01:23.476555 22259 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.1:42641
I20260807 23:01:23.476603 22469 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.1:42641 every 8 connection(s)
I20260807 23:01:23.478048 22259 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786143361779964-25976-0/minicluster-data/ts-0/data/info.pb
I20260807 23:01:23.481231 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 22259
I20260807 23:01:23.481482 25976 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786143361779964-25976-0/minicluster-data/ts-0/wal/instance
I20260807 23:01:23.482676 25976 external_mini_cluster.cc:1428] Setting key 8181dc5b61b172300b4690e369f7cf5c
Aug 07 23:01:23 dist-test-slave-z8x9 krb5kdc[22086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143683, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.1@KRBTEST.COM for kudu/127.25.94.62@KRBTEST.COM
I20260807 23:01:23.502549 22473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:23.480799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.1:48541 (local address 127.25.94.62:39399)
0807 23:01:23.481342 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0807 23:01:23.481349 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 23:01:23.482810 (+ 1461us) server_negotiation.cc:407] Connection header received
0807 23:01:23.485082 (+ 2272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:01:23.485086 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:01:23.485148 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:01:23.485300 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:01:23.487493 (+ 2193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:23.488187 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:23.490820 (+ 2633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:23.491227 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:23.491350 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:23.495821 (+ 4471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:01:23.495831 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:01:23.495838 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 23:01:23.495887 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:01:23.499021 (+ 3134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:01:23.499844 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:01:23.499853 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:01:23.499860 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:01:23.500014 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:01:23.500437 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:01:23.500445 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:01:23.500450 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:01:23.500839 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:01:23.501006 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0807 23:01:23.501777 (+ 771us) server_negotiation.cc:299] Negotiation successful
0807 23:01:23.502046 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":242,"threads_started":1}
I20260807 23:01:23.503130 22474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:23.481410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:39399 (local address 127.25.94.1:48541)
0807 23:01:23.482534 (+ 1124us) negotiation.cc:107] Waiting for socket to connect
0807 23:01:23.482610 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0807 23:01:23.484770 (+ 2160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 23:01:23.485580 (+ 810us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 23:01:23.485592 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 23:01:23.486422 (+ 830us) client_negotiation.cc:189] Negotiated authn=SASL
0807 23:01:23.487305 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:01:23.487336 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:01:23.488369 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:01:23.488376 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:01:23.490649 (+ 2273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:01:23.490668 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:01:23.491470 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:01:23.491480 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:01:23.491832 (+ 352us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:23.492610 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 23:01:23.492651 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0807 23:01:23.495565 (+ 2914us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 23:01:23.499172 (+ 3607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 23:01:23.499184 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 23:01:23.499205 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0807 23:01:23.499691 (+ 486us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 23:01:23.500129 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 23:01:23.500133 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 23:01:23.500135 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 23:01:23.500312 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 23:01:23.500960 (+ 648us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 23:01:23.500966 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 23:01:23.501360 (+ 394us) client_negotiation.cc:769] Sending connection context
0807 23:01:23.501588 (+ 228us) client_negotiation.cc:240] Negotiation successful
0807 23:01:23.501988 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":756,"thread_start_us":489,"threads_started":1}
I20260807 23:01:23.504745 22470 heartbeater.cc:347] Connected to a master server at 127.25.94.62:39399
I20260807 23:01:23.505071 22470 heartbeater.cc:464] Registering TS with master...
I20260807 23:01:23.505859 22470 heartbeater.cc:511] Master 127.25.94.62:39399 requested a full tablet report, sending...
I20260807 23:01:23.508040 22164 ts_manager.cc:194] Registered new tserver with Master: 49bb149190364633b51b66a547f21af1 (127.25.94.1:42641)
I20260807 23:01:23.509749 22164 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.94.1@KRBTEST.COM'} at 127.25.94.1:48541
WARNING: no policy specified for kudu/127.25.94.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.2.keytab.
Entry for principal kudu/127.25.94.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.2.keytab.
I20260807 23:01:23.552692 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786143361779964-25976-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786143361779964-25976-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786143361779964-25976-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786143361779964-25976-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.2:0
--local_ip_for_outbound_sockets=127.25.94.2
--webserver_interface=127.25.94.2
--webserver_port=0
--tserver_master_addrs=127.25.94.62:39399
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.2.keytab
--principal=kudu/127.25.94.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.20:38015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786143361779964-25976-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786143361779964-25976-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786143361779964-25976-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 23:01:23.908020 22480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 23:01:23.908326 22480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 23:01:23.908490 22480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 23:01:23.916092 22480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 23:01:23.916270 22480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 23:01:23.916443 22480 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.2
I20260807 23:01:23.926447 22480 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:38015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786143361779964-25976-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786143361779964-25976-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.2.keytab
--principal=kudu/127.25.94.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786143361779964-25976-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.94.2
--webserver_port=0
--tserver_master_addrs=127.25.94.62:39399
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.2
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786143361779964-25976-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 23:01:23.927786 22480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 23:01:23.929082 22480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 23:01:23.942493 22489 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 23:01:23.942458 22491 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 23:01:23.942426 22494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 23:01:23.943550 22480 server_base.cc:1034] running on GCE node
I20260807 23:01:23.944100 22480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 23:01:23.945005 22480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 23:01:23.946254 22480 hybrid_clock.cc:648] HybridClock initialized: now 1786143683946181 us; error 75 us; skew 500 ppm
I20260807 23:01:23.946653 22480 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 23:01:23 dist-test-slave-z8x9 krb5kdc[22086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143683, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 23:01:23.950024 22480 init.cc:377] Logged in from keytab as kudu/127.25.94.2@KRBTEST.COM (short username kudu)
I20260807 23:01:23.951642 22480 webserver.cc:460] Webserver started at http://127.25.94.2:45901/ using document root <none> and password file <none>
I20260807 23:01:23.952355 22480 fs_manager.cc:359] Metadata directory not provided
I20260807 23:01:23.952507 22480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 23:01:23.952836 22480 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 23:01:23.955788 22480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786143361779964-25976-0/minicluster-data/ts-1/data/instance:
uuid: "0d938c2942394a81b990385241af6892"
format_stamp: "Formatted at 2026-08-07 23:01:23 on dist-test-slave-z8x9"
server_key: "7b3e90b92656a2c5212aca6f0d4cf692"
server_key_iv: "806373ef8e9b49affda203294fcc029b"
server_key_version: "encryptionkey@0"
I20260807 23:01:23.956561 22480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786143361779964-25976-0/minicluster-data/ts-1/wal/instance:
uuid: "0d938c2942394a81b990385241af6892"
format_stamp: "Formatted at 2026-08-07 23:01:23 on dist-test-slave-z8x9"
server_key: "7b3e90b92656a2c5212aca6f0d4cf692"
server_key_iv: "806373ef8e9b49affda203294fcc029b"
server_key_version: "encryptionkey@0"
I20260807 23:01:23.961802 22480 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260807 23:01:23.965288 22503 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 23:01:23.966544 22480 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 23:01:23.966730 22480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786143361779964-25976-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786143361779964-25976-0/minicluster-data/ts-1/wal
uuid: "0d938c2942394a81b990385241af6892"
format_stamp: "Formatted at 2026-08-07 23:01:23 on dist-test-slave-z8x9"
server_key: "7b3e90b92656a2c5212aca6f0d4cf692"
server_key_iv: "806373ef8e9b49affda203294fcc029b"
server_key_version: "encryptionkey@0"
I20260807 23:01:23.966946 22480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786143361779964-25976-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786143361779964-25976-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786143361779964-25976-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 23:01:23.979875 22480 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 23:01:23.983788 22480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 23:01:23.984088 22480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 23:01:23.985059 22480 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 23:01:23.986480 22480 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 23:01:23.986588 22480 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 23:01:23.986740 22480 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 23:01:23.986821 22480 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 23:01:24.022255 22480 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.2:34353
I20260807 23:01:24.022324 22685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.2:34353 every 8 connection(s)
I20260807 23:01:24.023579 22480 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786143361779964-25976-0/minicluster-data/ts-1/data/info.pb
I20260807 23:01:24.033609 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 22480
I20260807 23:01:24.033819 25976 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786143361779964-25976-0/minicluster-data/ts-1/wal/instance
I20260807 23:01:24.034554 25976 external_mini_cluster.cc:1428] Setting key 5114ba930c7c88ef0b00e0452766dcb8
Aug 07 23:01:24 dist-test-slave-z8x9 krb5kdc[22086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143683, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.2@KRBTEST.COM for kudu/127.25.94.62@KRBTEST.COM
I20260807 23:01:24.049310 22688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:24.026435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.2:52797 (local address 127.25.94.62:39399)
0807 23:01:24.026972 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0807 23:01:24.026977 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 23:01:24.028007 (+ 1030us) server_negotiation.cc:407] Connection header received
0807 23:01:24.029999 (+ 1992us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:01:24.030014 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:01:24.030114 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:01:24.030251 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:01:24.031844 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:24.032600 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:24.034474 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:24.035171 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:24.035315 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:24.040648 (+ 5333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:01:24.040662 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:01:24.040671 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 23:01:24.040751 (+ 80us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:01:24.045075 (+ 4324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:01:24.046208 (+ 1133us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:01:24.046217 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:01:24.046223 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:01:24.046406 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:01:24.047006 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:01:24.047014 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:01:24.047020 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:01:24.047463 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:01:24.047627 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0807 23:01:24.048476 (+ 849us) server_negotiation.cc:299] Negotiation successful
0807 23:01:24.048785 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":263,"threads_started":1}
I20260807 23:01:24.050064 22690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:24.027012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:39399 (local address 127.25.94.2:52797)
0807 23:01:24.027807 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0807 23:01:24.027851 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0807 23:01:24.029678 (+ 1827us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 23:01:24.030490 (+ 812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 23:01:24.030498 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 23:01:24.031044 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0807 23:01:24.031646 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:01:24.031664 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:01:24.032760 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:01:24.032764 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:01:24.034282 (+ 1518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:01:24.034296 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:01:24.035486 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:01:24.035504 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:01:24.035753 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:24.036756 (+ 1003us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 23:01:24.036807 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0807 23:01:24.040137 (+ 3330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 23:01:24.045362 (+ 5225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 23:01:24.045376 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 23:01:24.045400 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0807 23:01:24.046060 (+ 660us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 23:01:24.046593 (+ 533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 23:01:24.046600 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 23:01:24.046604 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 23:01:24.046844 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 23:01:24.047626 (+ 782us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 23:01:24.047633 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 23:01:24.048034 (+ 401us) client_negotiation.cc:769] Sending connection context
0807 23:01:24.048317 (+ 283us) client_negotiation.cc:240] Negotiation successful
0807 23:01:24.048766 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":318,"threads_started":1}
I20260807 23:01:24.051620 22686 heartbeater.cc:347] Connected to a master server at 127.25.94.62:39399
I20260807 23:01:24.051996 22686 heartbeater.cc:464] Registering TS with master...
I20260807 23:01:24.052788 22686 heartbeater.cc:511] Master 127.25.94.62:39399 requested a full tablet report, sending...
I20260807 23:01:24.054402 22164 ts_manager.cc:194] Registered new tserver with Master: 0d938c2942394a81b990385241af6892 (127.25.94.2:34353)
I20260807 23:01:24.055696 22164 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.94.2@KRBTEST.COM'} at 127.25.94.2:52797
WARNING: no policy specified for kudu/127.25.94.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.3.keytab.
Entry for principal kudu/127.25.94.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.3.keytab.
I20260807 23:01:24.109483 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786143361779964-25976-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786143361779964-25976-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786143361779964-25976-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786143361779964-25976-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.3:0
--local_ip_for_outbound_sockets=127.25.94.3
--webserver_interface=127.25.94.3
--webserver_port=0
--tserver_master_addrs=127.25.94.62:39399
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.3.keytab
--principal=kudu/127.25.94.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.20:38015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786143361779964-25976-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786143361779964-25976-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786143361779964-25976-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 23:01:24.465976 22695 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 23:01:24.466351 22695 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 23:01:24.466521 22695 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 23:01:24.474298 22695 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 23:01:24.474457 22695 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 23:01:24.474639 22695 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.3
I20260807 23:01:24.485081 22695 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:38015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786143361779964-25976-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786143361779964-25976-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.3.keytab
--principal=kudu/127.25.94.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786143361779964-25976-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.94.3
--webserver_port=0
--tserver_master_addrs=127.25.94.62:39399
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.3
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786143361779964-25976-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 23:01:24.486511 22695 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 23:01:24.487859 22695 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 23:01:24.501593 22708 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 23:01:24.501668 22705 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 23:01:24.501668 22706 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 23:01:24.503245 22695 server_base.cc:1034] running on GCE node
I20260807 23:01:24.504009 22695 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 23:01:24.504989 22695 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 23:01:24.506224 22695 hybrid_clock.cc:648] HybridClock initialized: now 1786143684506178 us; error 90 us; skew 500 ppm
I20260807 23:01:24.506608 22695 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 23:01:24 dist-test-slave-z8x9 krb5kdc[22086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143684, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 23:01:24.510109 22695 init.cc:377] Logged in from keytab as kudu/127.25.94.3@KRBTEST.COM (short username kudu)
I20260807 23:01:24.512596 22695 webserver.cc:460] Webserver started at http://127.25.94.3:40975/ using document root <none> and password file <none>
I20260807 23:01:24.512795 22470 heartbeater.cc:503] Master 127.25.94.62:39399 was elected leader, sending a full tablet report...
I20260807 23:01:24.513540 22695 fs_manager.cc:359] Metadata directory not provided
I20260807 23:01:24.513693 22695 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 23:01:24.514050 22695 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 23:01:24.517251 22695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786143361779964-25976-0/minicluster-data/ts-2/data/instance:
uuid: "b445f9e773ce4d4f80641b978f092089"
format_stamp: "Formatted at 2026-08-07 23:01:24 on dist-test-slave-z8x9"
server_key: "ef3aa9bbd0554f25ef37f8d50fcf7934"
server_key_iv: "1ad7df3ff59a2b610f66f2292f6b55cc"
server_key_version: "encryptionkey@0"
I20260807 23:01:24.518116 22695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786143361779964-25976-0/minicluster-data/ts-2/wal/instance:
uuid: "b445f9e773ce4d4f80641b978f092089"
format_stamp: "Formatted at 2026-08-07 23:01:24 on dist-test-slave-z8x9"
server_key: "ef3aa9bbd0554f25ef37f8d50fcf7934"
server_key_iv: "1ad7df3ff59a2b610f66f2292f6b55cc"
server_key_version: "encryptionkey@0"
I20260807 23:01:24.523795 22695 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260807 23:01:24.527513 22720 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 23:01:24.529023 22695 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 23:01:24.529261 22695 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786143361779964-25976-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786143361779964-25976-0/minicluster-data/ts-2/wal
uuid: "b445f9e773ce4d4f80641b978f092089"
format_stamp: "Formatted at 2026-08-07 23:01:24 on dist-test-slave-z8x9"
server_key: "ef3aa9bbd0554f25ef37f8d50fcf7934"
server_key_iv: "1ad7df3ff59a2b610f66f2292f6b55cc"
server_key_version: "encryptionkey@0"
I20260807 23:01:24.529497 22695 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786143361779964-25976-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786143361779964-25976-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786143361779964-25976-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 23:01:24.559438 22695 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 23:01:24.564116 22695 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 23:01:24.564433 22695 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 23:01:24.565516 22695 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 23:01:24.566871 22695 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 23:01:24.566982 22695 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 23:01:24.567138 22695 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 23:01:24.567216 22695 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 23:01:24.603508 22695 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.3:41229
I20260807 23:01:24.603602 22900 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.3:41229 every 8 connection(s)
I20260807 23:01:24.604907 22695 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786143361779964-25976-0/minicluster-data/ts-2/data/info.pb
I20260807 23:01:24.612932 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 22695
I20260807 23:01:24.613212 25976 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786143361779964-25976-0/minicluster-data/ts-2/wal/instance
I20260807 23:01:24.613684 25976 external_mini_cluster.cc:1428] Setting key c5108391fa7f650fc51dd2ff25e5531e
Aug 07 23:01:24 dist-test-slave-z8x9 krb5kdc[22086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143684, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.3@KRBTEST.COM for kudu/127.25.94.62@KRBTEST.COM
I20260807 23:01:24.625715 22906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:24.607731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.3:59921 (local address 127.25.94.62:39399)
0807 23:01:24.608265 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0807 23:01:24.608270 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 23:01:24.609046 (+ 776us) server_negotiation.cc:407] Connection header received
0807 23:01:24.611103 (+ 2057us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:01:24.611110 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:01:24.611229 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:01:24.611358 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:01:24.613008 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:24.613996 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:24.615986 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:24.616534 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:24.616664 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:24.620384 (+ 3720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:01:24.620389 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:01:24.620392 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 23:01:24.620425 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:01:24.623039 (+ 2614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:01:24.623747 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:01:24.623752 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:01:24.623754 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:01:24.623882 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:01:24.624249 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:01:24.624253 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:01:24.624255 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:01:24.624502 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:01:24.624618 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0807 23:01:24.625280 (+ 662us) server_negotiation.cc:299] Negotiation successful
0807 23:01:24.625442 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":271,"threads_started":1}
I20260807 23:01:24.626708 22907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:24.608094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:39399 (local address 127.25.94.3:59921)
0807 23:01:24.608868 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0807 23:01:24.608914 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0807 23:01:24.610807 (+ 1893us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 23:01:24.611619 (+ 812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 23:01:24.611628 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 23:01:24.612207 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0807 23:01:24.612814 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:01:24.612832 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:01:24.614157 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:01:24.614161 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:01:24.615803 (+ 1642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:01:24.615815 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:01:24.616822 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:01:24.616831 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:01:24.617097 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:24.617935 (+ 838us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 23:01:24.617963 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0807 23:01:24.620159 (+ 2196us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 23:01:24.623164 (+ 3005us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 23:01:24.623170 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 23:01:24.623183 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0807 23:01:24.623645 (+ 462us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 23:01:24.623974 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 23:01:24.623977 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 23:01:24.623980 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 23:01:24.624158 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 23:01:24.624608 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 23:01:24.624613 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 23:01:24.624915 (+ 302us) client_negotiation.cc:769] Sending connection context
0807 23:01:24.625148 (+ 233us) client_negotiation.cc:240] Negotiation successful
0807 23:01:24.625615 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":324,"threads_started":1}
I20260807 23:01:24.628510 22901 heartbeater.cc:347] Connected to a master server at 127.25.94.62:39399
I20260807 23:01:24.628892 22901 heartbeater.cc:464] Registering TS with master...
I20260807 23:01:24.629767 22901 heartbeater.cc:511] Master 127.25.94.62:39399 requested a full tablet report, sending...
I20260807 23:01:24.631165 22161 ts_manager.cc:194] Registered new tserver with Master: b445f9e773ce4d4f80641b978f092089 (127.25.94.3:41229)
I20260807 23:01:24.632030 22161 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.94.3@KRBTEST.COM'} at 127.25.94.3:59921
I20260807 23:01:24.642122 25976 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 23:01:24.659421 22906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:24.646900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43338 (local address 127.25.94.62:39399)
0807 23:01:24.647086 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0807 23:01:24.647091 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 23:01:24.647569 (+ 478us) server_negotiation.cc:407] Connection header received
0807 23:01:24.647696 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:01:24.647700 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:01:24.647767 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:01:24.647849 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:01:24.649338 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:24.650099 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:24.651391 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:24.651792 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:24.651888 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:24.654355 (+ 2467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:01:24.654360 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:01:24.654363 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 23:01:24.654394 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:01:24.657075 (+ 2681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:01:24.657727 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:01:24.657731 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:01:24.657733 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:01:24.657861 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:01:24.658176 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:01:24.658179 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:01:24.658181 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:01:24.658539 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:01:24.658716 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0807 23:01:24.659011 (+ 295us) server_negotiation.cc:299] Negotiation successful
0807 23:01:24.659173 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260807 23:01:24.664852 22161 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43338:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 23:01:24.672942 22161 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 23:01:24.713958 22926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:24.686141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.2:34353 (local address 127.0.0.1:54896)
0807 23:01:24.702592 (+ 16451us) negotiation.cc:107] Waiting for socket to connect
0807 23:01:24.702658 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0807 23:01:24.702944 (+ 286us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 23:01:24.706102 (+ 3158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 23:01:24.706114 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 23:01:24.706162 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 23:01:24.706759 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:01:24.706774 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:01:24.708708 (+ 1934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:01:24.708715 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:01:24.711245 (+ 2530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:01:24.711263 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:01:24.712872 (+ 1609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:01:24.712880 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:01:24.713080 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:24.713142 (+ 62us) client_negotiation.cc:769] Sending connection context
0807 23:01:24.713317 (+ 175us) client_negotiation.cc:240] Negotiation successful
0807 23:01:24.713539 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16033,"thread_start_us":17067,"threads_started":1}
I20260807 23:01:24.714697 22927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:24.686432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54896 (local address 127.25.94.2:34353)
0807 23:01:24.705221 (+ 18789us) server_negotiation.cc:206] Beginning negotiation
0807 23:01:24.705242 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0807 23:01:24.705283 (+ 41us) server_negotiation.cc:407] Connection header received
0807 23:01:24.705415 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:01:24.705427 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:01:24.705761 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:01:24.707018 (+ 1257us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 23:01:24.707262 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:24.708564 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:24.711442 (+ 2878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:24.712587 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:24.712798 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:24.713018 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0807 23:01:24.714215 (+ 1197us) server_negotiation.cc:299] Negotiation successful
0807 23:01:24.714400 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18557,"thread_start_us":18435,"threads_started":1}
I20260807 23:01:24.714787 22930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:24.686487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.1:42641 (local address 127.0.0.1:45484)
0807 23:01:24.706274 (+ 19787us) negotiation.cc:107] Waiting for socket to connect
0807 23:01:24.706305 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0807 23:01:24.706568 (+ 263us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 23:01:24.707501 (+ 933us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 23:01:24.707505 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 23:01:24.707530 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 23:01:24.707996 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:01:24.708015 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:01:24.709523 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:01:24.709530 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:01:24.711686 (+ 2156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:01:24.711700 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:01:24.713922 (+ 2222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:01:24.713930 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:01:24.714124 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:24.714166 (+ 42us) client_negotiation.cc:769] Sending connection context
0807 23:01:24.714256 (+ 90us) client_negotiation.cc:240] Negotiation successful
0807 23:01:24.714393 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19607,"mutex_wait_us":1,"thread_start_us":19873,"threads_started":1}
I20260807 23:01:24.715420 22931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:24.687189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45484 (local address 127.25.94.1:42641)
0807 23:01:24.704664 (+ 17475us) server_negotiation.cc:206] Beginning negotiation
0807 23:01:24.704679 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0807 23:01:24.706430 (+ 1751us) server_negotiation.cc:407] Connection header received
0807 23:01:24.706787 (+ 357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:01:24.706796 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:01:24.707033 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:01:24.707200 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 23:01:24.708194 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:24.709391 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:24.711856 (+ 2465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:24.713440 (+ 1584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:24.713555 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:24.713764 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0807 23:01:24.714996 (+ 1232us) server_negotiation.cc:299] Negotiation successful
0807 23:01:24.715113 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17241,"thread_start_us":17017,"threads_started":1}
I20260807 23:01:24.718189 22928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:24.686880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.3:41229 (local address 127.0.0.1:35784)
0807 23:01:24.706719 (+ 19839us) negotiation.cc:107] Waiting for socket to connect
0807 23:01:24.706738 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 23:01:24.706859 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 23:01:24.707832 (+ 973us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 23:01:24.707836 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 23:01:24.707866 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 23:01:24.708299 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:01:24.708308 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:01:24.710306 (+ 1998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:01:24.710311 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:01:24.712434 (+ 2123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:01:24.712449 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:01:24.717354 (+ 4905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:01:24.717361 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:01:24.717546 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:24.717581 (+ 35us) client_negotiation.cc:769] Sending connection context
0807 23:01:24.717645 (+ 64us) client_negotiation.cc:240] Negotiation successful
0807 23:01:24.717765 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19672,"mutex_wait_us":61,"thread_start_us":19055,"threads_started":1}
I20260807 23:01:24.719096 22932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:24.687441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35784 (local address 127.25.94.3:41229)
0807 23:01:24.705685 (+ 18244us) server_negotiation.cc:206] Beginning negotiation
0807 23:01:24.705698 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0807 23:01:24.706994 (+ 1296us) server_negotiation.cc:407] Connection header received
0807 23:01:24.707106 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:01:24.707113 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:01:24.707315 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:01:24.707507 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 23:01:24.708498 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:24.710121 (+ 1623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:24.712608 (+ 2487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:24.717062 (+ 4454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:24.718499 (+ 1437us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:24.718674 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0807 23:01:24.718734 (+ 60us) server_negotiation.cc:299] Negotiation successful
0807 23:01:24.718852 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18015,"thread_start_us":17832,"threads_started":1}
I20260807 23:01:24.720000 22363 tablet_service.cc:1467] Processing CreateTablet for tablet 8ae563a5b83145068ee22350f8c327f9 (DEFAULT_TABLE table=test-table [id=2ff232a4cf0d415db3da2b75afe8ce95]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 23:01:24.720002 22583 tablet_service.cc:1467] Processing CreateTablet for tablet 8ae563a5b83145068ee22350f8c327f9 (DEFAULT_TABLE table=test-table [id=2ff232a4cf0d415db3da2b75afe8ce95]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 23:01:24.721520 22363 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ae563a5b83145068ee22350f8c327f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 23:01:24.721517 22583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ae563a5b83145068ee22350f8c327f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 23:01:24.722312 22794 tablet_service.cc:1467] Processing CreateTablet for tablet 8ae563a5b83145068ee22350f8c327f9 (DEFAULT_TABLE table=test-table [id=2ff232a4cf0d415db3da2b75afe8ce95]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 23:01:24.723541 22794 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ae563a5b83145068ee22350f8c327f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 23:01:24.735350 22934 tablet_bootstrap.cc:492] T 8ae563a5b83145068ee22350f8c327f9 P b445f9e773ce4d4f80641b978f092089: Bootstrap starting.
I20260807 23:01:24.737957 22936 tablet_bootstrap.cc:492] T 8ae563a5b83145068ee22350f8c327f9 P 0d938c2942394a81b990385241af6892: Bootstrap starting.
I20260807 23:01:24.740365 22934 tablet_bootstrap.cc:654] T 8ae563a5b83145068ee22350f8c327f9 P b445f9e773ce4d4f80641b978f092089: Neither blocks nor log segments found. Creating new log.
I20260807 23:01:24.741340 22935 tablet_bootstrap.cc:492] T 8ae563a5b83145068ee22350f8c327f9 P 49bb149190364633b51b66a547f21af1: Bootstrap starting.
I20260807 23:01:24.741777 22934 log.cc:824] T 8ae563a5b83145068ee22350f8c327f9 P b445f9e773ce4d4f80641b978f092089: Log is configured to *not* fsync() on all Append() calls
I20260807 23:01:24.742393 22936 tablet_bootstrap.cc:654] T 8ae563a5b83145068ee22350f8c327f9 P 0d938c2942394a81b990385241af6892: Neither blocks nor log segments found. Creating new log.
I20260807 23:01:24.743845 22936 log.cc:824] T 8ae563a5b83145068ee22350f8c327f9 P 0d938c2942394a81b990385241af6892: Log is configured to *not* fsync() on all Append() calls
I20260807 23:01:24.744709 22935 tablet_bootstrap.cc:654] T 8ae563a5b83145068ee22350f8c327f9 P 49bb149190364633b51b66a547f21af1: Neither blocks nor log segments found. Creating new log.
I20260807 23:01:24.745306 22934 tablet_bootstrap.cc:492] T 8ae563a5b83145068ee22350f8c327f9 P b445f9e773ce4d4f80641b978f092089: No bootstrap required, opened a new log
I20260807 23:01:24.745615 22934 ts_tablet_manager.cc:1397] T 8ae563a5b83145068ee22350f8c327f9 P b445f9e773ce4d4f80641b978f092089: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.002s
I20260807 23:01:24.746086 22935 log.cc:824] T 8ae563a5b83145068ee22350f8c327f9 P 49bb149190364633b51b66a547f21af1: Log is configured to *not* fsync() on all Append() calls
I20260807 23:01:24.747416 22936 tablet_bootstrap.cc:492] T 8ae563a5b83145068ee22350f8c327f9 P 0d938c2942394a81b990385241af6892: No bootstrap required, opened a new log
I20260807 23:01:24.747701 22936 ts_tablet_manager.cc:1397] T 8ae563a5b83145068ee22350f8c327f9 P 0d938c2942394a81b990385241af6892: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260807 23:01:24.749384 22935 tablet_bootstrap.cc:492] T 8ae563a5b83145068ee22350f8c327f9 P 49bb149190364633b51b66a547f21af1: No bootstrap required, opened a new log
I20260807 23:01:24.749682 22935 ts_tablet_manager.cc:1397] T 8ae563a5b83145068ee22350f8c327f9 P 49bb149190364633b51b66a547f21af1: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260807 23:01:24.754110 22934 raft_consensus.cc:348] T 8ae563a5b83145068ee22350f8c327f9 P b445f9e773ce4d4f80641b978f092089 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d938c2942394a81b990385241af6892" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 34353 } } peers { permanent_uuid: "49bb149190364633b51b66a547f21af1" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 42641 } } peers { permanent_uuid: "b445f9e773ce4d4f80641b978f092089" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 41229 } }
I20260807 23:01:24.754500 22934 raft_consensus.cc:374] T 8ae563a5b83145068ee22350f8c327f9 P b445f9e773ce4d4f80641b978f092089 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 23:01:24.754637 22934 raft_consensus.cc:729] T 8ae563a5b83145068ee22350f8c327f9 P b445f9e773ce4d4f80641b978f092089 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b445f9e773ce4d4f80641b978f092089, State: Initialized, Role: FOLLOWER
I20260807 23:01:24.755242 22934 consensus_queue.cc:260] T 8ae563a5b83145068ee22350f8c327f9 P b445f9e773ce4d4f80641b978f092089 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d938c2942394a81b990385241af6892" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 34353 } } peers { permanent_uuid: "49bb149190364633b51b66a547f21af1" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 42641 } } peers { permanent_uuid: "b445f9e773ce4d4f80641b978f092089" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 41229 } }
I20260807 23:01:24.756429 22901 heartbeater.cc:503] Master 127.25.94.62:39399 was elected leader, sending a full tablet report...
I20260807 23:01:24.757097 22934 ts_tablet_manager.cc:1428] T 8ae563a5b83145068ee22350f8c327f9 P b445f9e773ce4d4f80641b978f092089: Time spent starting tablet: real 0.011s user 0.005s sys 0.006s
I20260807 23:01:24.758409 22936 raft_consensus.cc:348] T 8ae563a5b83145068ee22350f8c327f9 P 0d938c2942394a81b990385241af6892 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d938c2942394a81b990385241af6892" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 34353 } } peers { permanent_uuid: "49bb149190364633b51b66a547f21af1" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 42641 } } peers { permanent_uuid: "b445f9e773ce4d4f80641b978f092089" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 41229 } }
I20260807 23:01:24.758873 22936 raft_consensus.cc:374] T 8ae563a5b83145068ee22350f8c327f9 P 0d938c2942394a81b990385241af6892 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 23:01:24.758966 22936 raft_consensus.cc:729] T 8ae563a5b83145068ee22350f8c327f9 P 0d938c2942394a81b990385241af6892 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d938c2942394a81b990385241af6892, State: Initialized, Role: FOLLOWER
I20260807 23:01:24.759533 22936 consensus_queue.cc:260] T 8ae563a5b83145068ee22350f8c327f9 P 0d938c2942394a81b990385241af6892 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d938c2942394a81b990385241af6892" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 34353 } } peers { permanent_uuid: "49bb149190364633b51b66a547f21af1" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 42641 } } peers { permanent_uuid: "b445f9e773ce4d4f80641b978f092089" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 41229 } }
I20260807 23:01:24.760865 22686 heartbeater.cc:503] Master 127.25.94.62:39399 was elected leader, sending a full tablet report...
I20260807 23:01:24.760689 22935 raft_consensus.cc:348] T 8ae563a5b83145068ee22350f8c327f9 P 49bb149190364633b51b66a547f21af1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d938c2942394a81b990385241af6892" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 34353 } } peers { permanent_uuid: "49bb149190364633b51b66a547f21af1" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 42641 } } peers { permanent_uuid: "b445f9e773ce4d4f80641b978f092089" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 41229 } }
I20260807 23:01:24.761102 22935 raft_consensus.cc:374] T 8ae563a5b83145068ee22350f8c327f9 P 49bb149190364633b51b66a547f21af1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 23:01:24.761253 22935 raft_consensus.cc:729] T 8ae563a5b83145068ee22350f8c327f9 P 49bb149190364633b51b66a547f21af1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bb149190364633b51b66a547f21af1, State: Initialized, Role: FOLLOWER
I20260807 23:01:24.761425 22936 ts_tablet_manager.cc:1428] T 8ae563a5b83145068ee22350f8c327f9 P 0d938c2942394a81b990385241af6892: Time spent starting tablet: real 0.014s user 0.007s sys 0.007s
I20260807 23:01:24.761816 22935 consensus_queue.cc:260] T 8ae563a5b83145068ee22350f8c327f9 P 49bb149190364633b51b66a547f21af1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d938c2942394a81b990385241af6892" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 34353 } } peers { permanent_uuid: "49bb149190364633b51b66a547f21af1" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 42641 } } peers { permanent_uuid: "b445f9e773ce4d4f80641b978f092089" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 41229 } }
I20260807 23:01:24.764081 22935 ts_tablet_manager.cc:1428] T 8ae563a5b83145068ee22350f8c327f9 P 49bb149190364633b51b66a547f21af1: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
W20260807 23:01:24.775650 22687 tablet.cc:2369] T 8ae563a5b83145068ee22350f8c327f9 P 0d938c2942394a81b990385241af6892: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 23:01:24.840080 22945 raft_consensus.cc:482] T 8ae563a5b83145068ee22350f8c327f9 P 49bb149190364633b51b66a547f21af1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 23:01:24.840478 22945 raft_consensus.cc:504] T 8ae563a5b83145068ee22350f8c327f9 P 49bb149190364633b51b66a547f21af1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d938c2942394a81b990385241af6892" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 34353 } } peers { permanent_uuid: "49bb149190364633b51b66a547f21af1" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 42641 } } peers { permanent_uuid: "b445f9e773ce4d4f80641b978f092089" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 41229 } }
I20260807 23:01:24.842682 22945 leader_election.cc:288] T 8ae563a5b83145068ee22350f8c327f9 P 49bb149190364633b51b66a547f21af1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d938c2942394a81b990385241af6892 (127.25.94.2:34353), b445f9e773ce4d4f80641b978f092089 (127.25.94.3:41229)
I20260807 23:01:24.850116 22932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:24.843337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.1:60821 (local address 127.25.94.3:41229)
0807 23:01:24.843580 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0807 23:01:24.843587 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 23:01:24.844223 (+ 636us) server_negotiation.cc:407] Connection header received
0807 23:01:24.844468 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:01:24.844472 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:01:24.844553 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:01:24.844649 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 23:01:24.845324 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:24.846252 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:24.848318 (+ 2066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:24.849089 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:24.849271 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:24.849399 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0807 23:01:24.849722 (+ 323us) server_negotiation.cc:299] Negotiation successful
0807 23:01:24.849858 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260807 23:01:24.850191 22947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:24.843039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.3:41229 (local address 127.25.94.1:60821)
0807 23:01:24.844112 (+ 1073us) negotiation.cc:107] Waiting for socket to connect
0807 23:01:24.844130 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 23:01:24.844318 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 23:01:24.844663 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 23:01:24.844667 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 23:01:24.844691 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 23:01:24.845150 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:01:24.845187 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:01:24.846381 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:01:24.846388 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:01:24.848157 (+ 1769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:01:24.848170 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:01:24.849330 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:01:24.849336 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:01:24.849562 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:24.849588 (+ 26us) client_negotiation.cc:769] Sending connection context
0807 23:01:24.849697 (+ 109us) client_negotiation.cc:240] Negotiation successful
0807 23:01:24.849821 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":902,"thread_start_us":528,"threads_started":1}
I20260807 23:01:24.850332 22927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:24.843109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.1:57395 (local address 127.25.94.2:34353)
0807 23:01:24.843339 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0807 23:01:24.843348 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 23:01:24.844337 (+ 989us) server_negotiation.cc:407] Connection header received
0807 23:01:24.844552 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:01:24.844556 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:01:24.844617 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:01:24.844718 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 23:01:24.845570 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:24.846546 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:24.848491 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:24.849331 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:24.849514 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:24.849636 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0807 23:01:24.849933 (+ 297us) server_negotiation.cc:299] Negotiation successful
0807 23:01:24.850055 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260807 23:01:24.850781 22948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:24.843082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.2:34353 (local address 127.25.94.1:57395)
0807 23:01:24.844215 (+ 1133us) negotiation.cc:107] Waiting for socket to connect
0807 23:01:24.844230 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 23:01:24.844391 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 23:01:24.844740 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 23:01:24.844744 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 23:01:24.844773 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 23:01:24.845375 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:01:24.845387 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:01:24.846681 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:01:24.846686 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:01:24.848336 (+ 1650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:01:24.848348 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:01:24.849609 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:01:24.849615 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:01:24.849778 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:24.849801 (+ 23us) client_negotiation.cc:769] Sending connection context
0807 23:01:24.849906 (+ 105us) client_negotiation.cc:240] Negotiation successful
0807 23:01:24.850035 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1016,"thread_start_us":781,"threads_started":1}
I20260807 23:01:24.851357 22827 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ae563a5b83145068ee22350f8c327f9" candidate_uuid: "49bb149190364633b51b66a547f21af1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b445f9e773ce4d4f80641b978f092089" is_pre_election: true
I20260807 23:01:24.851670 22615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ae563a5b83145068ee22350f8c327f9" candidate_uuid: "49bb149190364633b51b66a547f21af1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d938c2942394a81b990385241af6892" is_pre_election: true
I20260807 23:01:24.851927 22827 raft_consensus.cc:2454] T 8ae563a5b83145068ee22350f8c327f9 P b445f9e773ce4d4f80641b978f092089 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49bb149190364633b51b66a547f21af1 in term 0.
I20260807 23:01:24.852186 22615 raft_consensus.cc:2454] T 8ae563a5b83145068ee22350f8c327f9 P 0d938c2942394a81b990385241af6892 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49bb149190364633b51b66a547f21af1 in term 0.
I20260807 23:01:24.852717 22290 leader_election.cc:302] T 8ae563a5b83145068ee22350f8c327f9 P 49bb149190364633b51b66a547f21af1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49bb149190364633b51b66a547f21af1, b445f9e773ce4d4f80641b978f092089; no voters:
I20260807 23:01:24.853140 22945 raft_consensus.cc:2781] T 8ae563a5b83145068ee22350f8c327f9 P 49bb149190364633b51b66a547f21af1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 23:01:24.853351 22945 raft_consensus.cc:482] T 8ae563a5b83145068ee22350f8c327f9 P 49bb149190364633b51b66a547f21af1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 23:01:24.853466 22945 raft_consensus.cc:3037] T 8ae563a5b83145068ee22350f8c327f9 P 49bb149190364633b51b66a547f21af1 [term 0 FOLLOWER]: Advancing to term 1
I20260807 23:01:24.855695 22945 raft_consensus.cc:504] T 8ae563a5b83145068ee22350f8c327f9 P 49bb149190364633b51b66a547f21af1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d938c2942394a81b990385241af6892" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 34353 } } peers { permanent_uuid: "49bb149190364633b51b66a547f21af1" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 42641 } } peers { permanent_uuid: "b445f9e773ce4d4f80641b978f092089" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 41229 } }
I20260807 23:01:24.856369 22945 leader_election.cc:288] T 8ae563a5b83145068ee22350f8c327f9 P 49bb149190364633b51b66a547f21af1 [CANDIDATE]: Term 1 election: Requested vote from peers 0d938c2942394a81b990385241af6892 (127.25.94.2:34353), b445f9e773ce4d4f80641b978f092089 (127.25.94.3:41229)
W20260807 23:01:24.856729 22905 tablet.cc:2369] T 8ae563a5b83145068ee22350f8c327f9 P b445f9e773ce4d4f80641b978f092089: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 23:01:24.856906 22827 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ae563a5b83145068ee22350f8c327f9" candidate_uuid: "49bb149190364633b51b66a547f21af1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b445f9e773ce4d4f80641b978f092089"
I20260807 23:01:24.856949 22615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ae563a5b83145068ee22350f8c327f9" candidate_uuid: "49bb149190364633b51b66a547f21af1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d938c2942394a81b990385241af6892"
I20260807 23:01:24.857116 22827 raft_consensus.cc:3037] T 8ae563a5b83145068ee22350f8c327f9 P b445f9e773ce4d4f80641b978f092089 [term 0 FOLLOWER]: Advancing to term 1
I20260807 23:01:24.857197 22615 raft_consensus.cc:3037] T 8ae563a5b83145068ee22350f8c327f9 P 0d938c2942394a81b990385241af6892 [term 0 FOLLOWER]: Advancing to term 1
I20260807 23:01:24.859397 22615 raft_consensus.cc:2454] T 8ae563a5b83145068ee22350f8c327f9 P 0d938c2942394a81b990385241af6892 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49bb149190364633b51b66a547f21af1 in term 1.
I20260807 23:01:24.859455 22827 raft_consensus.cc:2454] T 8ae563a5b83145068ee22350f8c327f9 P b445f9e773ce4d4f80641b978f092089 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49bb149190364633b51b66a547f21af1 in term 1.
I20260807 23:01:24.859860 22288 leader_election.cc:302] T 8ae563a5b83145068ee22350f8c327f9 P 49bb149190364633b51b66a547f21af1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0d938c2942394a81b990385241af6892, 49bb149190364633b51b66a547f21af1, b445f9e773ce4d4f80641b978f092089; no voters:
I20260807 23:01:24.860257 22945 raft_consensus.cc:2781] T 8ae563a5b83145068ee22350f8c327f9 P 49bb149190364633b51b66a547f21af1 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 23:01:24.860710 22945 raft_consensus.cc:686] T 8ae563a5b83145068ee22350f8c327f9 P 49bb149190364633b51b66a547f21af1 [term 1 LEADER]: Becoming Leader. State: Replica: 49bb149190364633b51b66a547f21af1, State: Running, Role: LEADER
I20260807 23:01:24.861477 22945 consensus_queue.cc:237] T 8ae563a5b83145068ee22350f8c327f9 P 49bb149190364633b51b66a547f21af1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d938c2942394a81b990385241af6892" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 34353 } } peers { permanent_uuid: "49bb149190364633b51b66a547f21af1" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 42641 } } peers { permanent_uuid: "b445f9e773ce4d4f80641b978f092089" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 41229 } }
I20260807 23:01:24.866117 22161 catalog_manager.cc:5495] T 8ae563a5b83145068ee22350f8c327f9 P 49bb149190364633b51b66a547f21af1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 49bb149190364633b51b66a547f21af1 (127.25.94.1). New cstate: current_term: 1 leader_uuid: "49bb149190364633b51b66a547f21af1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d938c2942394a81b990385241af6892" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 34353 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49bb149190364633b51b66a547f21af1" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 42641 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b445f9e773ce4d4f80641b978f092089" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 41229 } health_report { overall_health: UNKNOWN } } }
I20260807 23:01:24.912400 22931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:24.907180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45488 (local address 127.25.94.1:42641)
0807 23:01:24.907407 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0807 23:01:24.907411 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 23:01:24.907450 (+ 39us) server_negotiation.cc:407] Connection header received
0807 23:01:24.907657 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:01:24.907661 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:01:24.907759 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:01:24.907888 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 23:01:24.908223 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:24.908970 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:24.910440 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:24.910834 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:24.910924 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:24.911378 (+ 454us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 23:01:24.911894 (+ 516us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 23:01:24.912000 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0807 23:01:24.912107 (+ 107us) server_negotiation.cc:299] Negotiation successful
0807 23:01:24.912173 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260807 23:01:24.920689 22827 raft_consensus.cc:1261] T 8ae563a5b83145068ee22350f8c327f9 P b445f9e773ce4d4f80641b978f092089 [term 1 FOLLOWER]: Refusing update from remote peer 49bb149190364633b51b66a547f21af1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 23:01:24.920686 22615 raft_consensus.cc:1261] T 8ae563a5b83145068ee22350f8c327f9 P 0d938c2942394a81b990385241af6892 [term 1 FOLLOWER]: Refusing update from remote peer 49bb149190364633b51b66a547f21af1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 23:01:24.921751 22949 consensus_queue.cc:1035] T 8ae563a5b83145068ee22350f8c327f9 P 49bb149190364633b51b66a547f21af1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d938c2942394a81b990385241af6892" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 34353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 23:01:24.922106 22945 consensus_queue.cc:1035] T 8ae563a5b83145068ee22350f8c327f9 P 49bb149190364633b51b66a547f21af1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b445f9e773ce4d4f80641b978f092089" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 41229 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 23:01:24.940630 22956 mvcc.cc:205] Tried to move back new op lower bound from 7316044533425197056 to 7316044533197348864. Current Snapshot: MvccSnapshot[applied={T|T < 7316044533425197056}]
I20260807 23:01:24.949332 22164 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43338:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 23:01:24.949659 22164 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43338:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 23:01:24.953861 22164 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ed2e823495994a278850048096d62cbc: Sending DeleteTablet for 3 replicas of tablet 8ae563a5b83145068ee22350f8c327f9
I20260807 23:01:24.955210 22794 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ae563a5b83145068ee22350f8c327f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 23:01:24 UTC) from {username='kudu', principal='kudu/127.25.94.62@KRBTEST.COM'} at 127.0.0.1:35784
I20260807 23:01:24.955205 22583 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ae563a5b83145068ee22350f8c327f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 23:01:24 UTC) from {username='kudu', principal='kudu/127.25.94.62@KRBTEST.COM'} at 127.0.0.1:54896
I20260807 23:01:24.955988 22363 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ae563a5b83145068ee22350f8c327f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 23:01:24 UTC) from {username='kudu', principal='kudu/127.25.94.62@KRBTEST.COM'} at 127.0.0.1:45484
I20260807 23:01:24.956318 22972 tablet_replica.cc:330] T 8ae563a5b83145068ee22350f8c327f9 P b445f9e773ce4d4f80641b978f092089: stopping tablet replica
I20260807 23:01:24.956810 22972 raft_consensus.cc:2229] T 8ae563a5b83145068ee22350f8c327f9 P b445f9e773ce4d4f80641b978f092089 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 23:01:24.957476 22972 raft_consensus.cc:2258] T 8ae563a5b83145068ee22350f8c327f9 P b445f9e773ce4d4f80641b978f092089 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 23:01:24.958074 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 22259
I20260807 23:01:24.959913 22972 ts_tablet_manager.cc:1905] T 8ae563a5b83145068ee22350f8c327f9 P b445f9e773ce4d4f80641b978f092089: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 23:01:24.969702 22972 ts_tablet_manager.cc:1918] T 8ae563a5b83145068ee22350f8c327f9 P b445f9e773ce4d4f80641b978f092089: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 23:01:24.970047 22972 log.cc:1196] T 8ae563a5b83145068ee22350f8c327f9 P b445f9e773ce4d4f80641b978f092089: Deleting WAL directory at /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786143361779964-25976-0/minicluster-data/ts-2/wal/wals/8ae563a5b83145068ee22350f8c327f9
I20260807 23:01:24.970857 22972 ts_tablet_manager.cc:1939] T 8ae563a5b83145068ee22350f8c327f9 P b445f9e773ce4d4f80641b978f092089: Deleting consensus metadata
I20260807 23:01:24.972456 22976 tablet_replica.cc:330] T 8ae563a5b83145068ee22350f8c327f9 P 0d938c2942394a81b990385241af6892: stopping tablet replica
I20260807 23:01:24.973415 22141 catalog_manager.cc:4841] TS b445f9e773ce4d4f80641b978f092089 (127.25.94.3:41229): tablet 8ae563a5b83145068ee22350f8c327f9 (table test-table [id=2ff232a4cf0d415db3da2b75afe8ce95]) successfully deleted
I20260807 23:01:24.972946 22976 raft_consensus.cc:2229] T 8ae563a5b83145068ee22350f8c327f9 P 0d938c2942394a81b990385241af6892 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 23:01:24.974112 22977 tablet_replica.cc:330] T 8ae563a5b83145068ee22350f8c327f9 P 49bb149190364633b51b66a547f21af1: stopping tablet replica
I20260807 23:01:24.974205 22976 raft_consensus.cc:2258] T 8ae563a5b83145068ee22350f8c327f9 P 0d938c2942394a81b990385241af6892 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 23:01:24.974653 22977 raft_consensus.cc:2229] T 8ae563a5b83145068ee22350f8c327f9 P 49bb149190364633b51b66a547f21af1 [term 1 LEADER]: Raft consensus shutting down.
I20260807 23:01:24.975399 22977 raft_consensus.cc:2258] T 8ae563a5b83145068ee22350f8c327f9 P 49bb149190364633b51b66a547f21af1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 23:01:24.976653 22976 ts_tablet_manager.cc:1905] T 8ae563a5b83145068ee22350f8c327f9 P 0d938c2942394a81b990385241af6892: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 23:01:24.977864 22977 ts_tablet_manager.cc:1905] T 8ae563a5b83145068ee22350f8c327f9 P 49bb149190364633b51b66a547f21af1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 23:01:24.984160 22976 ts_tablet_manager.cc:1918] T 8ae563a5b83145068ee22350f8c327f9 P 0d938c2942394a81b990385241af6892: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 23:01:24.984354 22976 log.cc:1196] T 8ae563a5b83145068ee22350f8c327f9 P 0d938c2942394a81b990385241af6892: Deleting WAL directory at /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786143361779964-25976-0/minicluster-data/ts-1/wal/wals/8ae563a5b83145068ee22350f8c327f9
I20260807 23:01:24.984894 22976 ts_tablet_manager.cc:1939] T 8ae563a5b83145068ee22350f8c327f9 P 0d938c2942394a81b990385241af6892: Deleting consensus metadata
I20260807 23:01:24.985663 22977 ts_tablet_manager.cc:1918] T 8ae563a5b83145068ee22350f8c327f9 P 49bb149190364633b51b66a547f21af1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
Aug 07 23:01:24 dist-test-slave-z8x9 krb5kdc[22086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143682, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.1@KRBTEST.COM
I20260807 23:01:24.985914 22977 log.cc:1196] T 8ae563a5b83145068ee22350f8c327f9 P 49bb149190364633b51b66a547f21af1: Deleting WAL directory at /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786143361779964-25976-0/minicluster-data/ts-0/wal/wals/8ae563a5b83145068ee22350f8c327f9
I20260807 23:01:24.986675 22977 ts_tablet_manager.cc:1939] T 8ae563a5b83145068ee22350f8c327f9 P 49bb149190364633b51b66a547f21af1: Deleting consensus metadata
I20260807 23:01:24.986960 22135 catalog_manager.cc:4841] TS 0d938c2942394a81b990385241af6892 (127.25.94.2:34353): tablet 8ae563a5b83145068ee22350f8c327f9 (table test-table [id=2ff232a4cf0d415db3da2b75afe8ce95]) successfully deleted
I20260807 23:01:24.988404 22132 catalog_manager.cc:4841] TS 49bb149190364633b51b66a547f21af1 (127.25.94.1:42641): tablet 8ae563a5b83145068ee22350f8c327f9 (table test-table [id=2ff232a4cf0d415db3da2b75afe8ce95]) successfully deleted
I20260807 23:01:24.994848 22931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:24.959205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45490 (local address 127.25.94.1:42641)
0807 23:01:24.959370 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0807 23:01:24.959377 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 23:01:24.973104 (+ 13727us) server_negotiation.cc:407] Connection header received
0807 23:01:24.973985 (+ 881us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:01:24.973992 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:01:24.974077 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:01:24.974160 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:01:24.976136 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:24.977744 (+ 1608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:24.980332 (+ 2588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:24.981125 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:24.981258 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:24.987302 (+ 6044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:01:24.987320 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:01:24.987339 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 23:01:24.987400 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:01:24.991672 (+ 4272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:01:24.992402 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:01:24.992415 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:01:24.992420 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:01:24.992642 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:01:24.993023 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:01:24.993029 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:01:24.993032 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:01:24.993526 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:01:24.993859 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0807 23:01:24.994214 (+ 355us) server_negotiation.cc:299] Negotiation successful
0807 23:01:24.994466 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260807 23:01:24.995620 22463 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 23:01:25.112757 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 22259
I20260807 23:01:25.136937 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 22480
Aug 07 23:01:25 dist-test-slave-z8x9 krb5kdc[22086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143682, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.2@KRBTEST.COM
I20260807 23:01:25.155344 22927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:25.138023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54904 (local address 127.25.94.2:34353)
0807 23:01:25.138203 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0807 23:01:25.138208 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 23:01:25.138227 (+ 19us) server_negotiation.cc:407] Connection header received
0807 23:01:25.138291 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:01:25.138295 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:01:25.138381 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:01:25.138484 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:01:25.140100 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:25.140996 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:25.142447 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:25.142865 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:25.142975 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:25.148439 (+ 5464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:01:25.148460 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:01:25.148477 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 23:01:25.148522 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:01:25.151820 (+ 3298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:01:25.152728 (+ 908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:01:25.152737 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:01:25.152739 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:01:25.152898 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:01:25.153633 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:01:25.153640 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:01:25.153644 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:01:25.154080 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:01:25.154403 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0807 23:01:25.154744 (+ 341us) server_negotiation.cc:299] Negotiation successful
0807 23:01:25.154991 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260807 23:01:25.156148 22681 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 23:01:25.251276 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 22480
I20260807 23:01:25.272926 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 22695
Aug 07 23:01:25 dist-test-slave-z8x9 krb5kdc[22086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143682, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.3@KRBTEST.COM
I20260807 23:01:25.289595 22932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:25.273833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35790 (local address 127.25.94.3:41229)
0807 23:01:25.274029 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0807 23:01:25.274034 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 23:01:25.274054 (+ 20us) server_negotiation.cc:407] Connection header received
0807 23:01:25.274114 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:01:25.274118 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:01:25.274194 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:01:25.274305 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:01:25.275785 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:25.276618 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:25.278220 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:25.278707 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:25.278826 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:25.283641 (+ 4815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:01:25.283660 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:01:25.283677 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 23:01:25.283718 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:01:25.286770 (+ 3052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:01:25.287653 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:01:25.287661 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:01:25.287663 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:01:25.287834 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:01:25.288185 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:01:25.288189 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:01:25.288190 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:01:25.288562 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:01:25.288786 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0807 23:01:25.289060 (+ 274us) server_negotiation.cc:299] Negotiation successful
0807 23:01:25.289319 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260807 23:01:25.290274 22895 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 23:01:25.383728 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 22695
I20260807 23:01:25.412724 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 22105
I20260807 23:01:25.414094 22209 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 23:01:25.526623 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 22105
2026-08-07T23:01:25Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4935 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-07T23:01:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T23:01:25Z Disabled control of system clock
I20260807 23:01:25.632066 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786143361779964-25976-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786143361779964-25976-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786143361779964-25976-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.62:42617
--webserver_interface=127.25.94.62
--webserver_port=0
--builtin_ntp_servers=127.25.94.20:34677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.94.62:42617
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260807 23:01:25.978289 22992 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 23:01:25.978610 22992 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 23:01:25.978696 22992 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 23:01:25.986430 22992 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 23:01:25.986569 22992 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 23:01:25.986608 22992 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 23:01:25.986660 22992 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 23:01:25.986706 22992 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 23:01:25.996232 22992 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:34677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786143361779964-25976-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786143361779964-25976-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.94.62:42617
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.62:42617
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.94.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786143361779964-25976-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 23:01:25.997684 22992 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 23:01:25.999517 22992 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 23:01:26.011488 23003 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 23:01:26.011499 22999 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 23:01:26.011489 22998 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 23:01:26.012523 22992 server_base.cc:1034] running on GCE node
I20260807 23:01:26.013522 22992 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 23:01:26.014933 22992 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 23:01:26.016140 22992 hybrid_clock.cc:648] HybridClock initialized: now 1786143686016075 us; error 67 us; skew 500 ppm
I20260807 23:01:26.016525 22992 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 23:01:26.019150 22992 webserver.cc:460] Webserver started at http://127.25.94.62:41421/ using document root <none> and password file <none>
I20260807 23:01:26.019884 22992 fs_manager.cc:359] Metadata directory not provided
I20260807 23:01:26.019979 22992 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 23:01:26.020336 22992 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 23:01:26.023339 22992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786143361779964-25976-0/minicluster-data/master-0/data/instance:
uuid: "26c9833d46fd47e1b35c1341716809ac"
format_stamp: "Formatted at 2026-08-07 23:01:26 on dist-test-slave-z8x9"
server_key: "b14fa4b91c75e9e7544a19caf379439f"
server_key_iv: "692bcc5a24d7557337828ddd8f5b1dc7"
server_key_version: "encryptionkey@0"
I20260807 23:01:26.024173 22992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786143361779964-25976-0/minicluster-data/master-0/wal/instance:
uuid: "26c9833d46fd47e1b35c1341716809ac"
format_stamp: "Formatted at 2026-08-07 23:01:26 on dist-test-slave-z8x9"
server_key: "b14fa4b91c75e9e7544a19caf379439f"
server_key_iv: "692bcc5a24d7557337828ddd8f5b1dc7"
server_key_version: "encryptionkey@0"
I20260807 23:01:26.029627 22992 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260807 23:01:26.033135 23011 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 23:01:26.034678 22992 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 23:01:26.034843 22992 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786143361779964-25976-0/minicluster-data/master-0/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786143361779964-25976-0/minicluster-data/master-0/wal
uuid: "26c9833d46fd47e1b35c1341716809ac"
format_stamp: "Formatted at 2026-08-07 23:01:26 on dist-test-slave-z8x9"
server_key: "b14fa4b91c75e9e7544a19caf379439f"
server_key_iv: "692bcc5a24d7557337828ddd8f5b1dc7"
server_key_version: "encryptionkey@0"
I20260807 23:01:26.035028 22992 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786143361779964-25976-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786143361779964-25976-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786143361779964-25976-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 23:01:26.049451 22992 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 23:01:26.050267 22992 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 23:01:26.050585 22992 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 23:01:26.069983 22992 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.62:42617
I20260807 23:01:26.070004 23100 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.62:42617 every 8 connection(s)
I20260807 23:01:26.071417 22992 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
I20260807 23:01:26.075358 23101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 23:01:26.080212 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 22992
I20260807 23:01:26.080443 25976 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786143361779964-25976-0/minicluster-data/master-0/wal/instance
I20260807 23:01:26.080941 25976 external_mini_cluster.cc:1428] Setting key 9b658e93365fc3cd7e6033e0d95369b5
I20260807 23:01:26.087602 23101 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26c9833d46fd47e1b35c1341716809ac: Bootstrap starting.
I20260807 23:01:26.091275 23101 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 26c9833d46fd47e1b35c1341716809ac: Neither blocks nor log segments found. Creating new log.
I20260807 23:01:26.092473 23101 log.cc:824] T 00000000000000000000000000000000 P 26c9833d46fd47e1b35c1341716809ac: Log is configured to *not* fsync() on all Append() calls
I20260807 23:01:26.095906 23104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:26.083260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51996 (local address 127.25.94.62:42617)
0807 23:01:26.084379 (+ 1119us) server_negotiation.cc:206] Beginning negotiation
0807 23:01:26.084390 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 23:01:26.084433 (+ 43us) server_negotiation.cc:407] Connection header received
0807 23:01:26.086322 (+ 1889us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:01:26.086348 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:01:26.086823 (+ 475us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:01:26.087234 (+ 411us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:01:26.087824 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:26.089148 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:26.090713 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:26.091511 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:26.091720 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:26.093416 (+ 1696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:01:26.093436 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:01:26.093457 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 23:01:26.094066 (+ 609us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:01:26.094151 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 23:01:26.094173 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:01:26.094335 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0807 23:01:26.094600 (+ 265us) server_negotiation.cc:299] Negotiation successful
0807 23:01:26.094946 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":696,"thread_start_us":441,"threads_started":1}
I20260807 23:01:26.096107 23101 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26c9833d46fd47e1b35c1341716809ac: No bootstrap required, opened a new log
I20260807 23:01:26.105865 23101 raft_consensus.cc:348] T 00000000000000000000000000000000 P 26c9833d46fd47e1b35c1341716809ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26c9833d46fd47e1b35c1341716809ac" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 42617 } }
I20260807 23:01:26.106329 23101 raft_consensus.cc:374] T 00000000000000000000000000000000 P 26c9833d46fd47e1b35c1341716809ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 23:01:26.106455 23101 raft_consensus.cc:729] T 00000000000000000000000000000000 P 26c9833d46fd47e1b35c1341716809ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26c9833d46fd47e1b35c1341716809ac, State: Initialized, Role: FOLLOWER
I20260807 23:01:26.107259 23101 consensus_queue.cc:260] T 00000000000000000000000000000000 P 26c9833d46fd47e1b35c1341716809ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26c9833d46fd47e1b35c1341716809ac" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 42617 } }
I20260807 23:01:26.107558 23101 raft_consensus.cc:388] T 00000000000000000000000000000000 P 26c9833d46fd47e1b35c1341716809ac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 23:01:26.107746 23101 raft_consensus.cc:482] T 00000000000000000000000000000000 P 26c9833d46fd47e1b35c1341716809ac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 23:01:26.107957 23101 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 26c9833d46fd47e1b35c1341716809ac [term 0 FOLLOWER]: Advancing to term 1
I20260807 23:01:26.110345 23101 raft_consensus.cc:504] T 00000000000000000000000000000000 P 26c9833d46fd47e1b35c1341716809ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26c9833d46fd47e1b35c1341716809ac" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 42617 } }
I20260807 23:01:26.110917 23101 leader_election.cc:302] T 00000000000000000000000000000000 P 26c9833d46fd47e1b35c1341716809ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 26c9833d46fd47e1b35c1341716809ac; no voters:
I20260807 23:01:26.111589 23101 leader_election.cc:288] T 00000000000000000000000000000000 P 26c9833d46fd47e1b35c1341716809ac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 23:01:26.111745 23108 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 26c9833d46fd47e1b35c1341716809ac [term 1 FOLLOWER]: Leader election won for term 1
I20260807 23:01:26.112607 23108 raft_consensus.cc:686] T 00000000000000000000000000000000 P 26c9833d46fd47e1b35c1341716809ac [term 1 LEADER]: Becoming Leader. State: Replica: 26c9833d46fd47e1b35c1341716809ac, State: Running, Role: LEADER
I20260807 23:01:26.113245 23108 consensus_queue.cc:237] T 00000000000000000000000000000000 P 26c9833d46fd47e1b35c1341716809ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26c9833d46fd47e1b35c1341716809ac" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 42617 } }
I20260807 23:01:26.113991 23101 sys_catalog.cc:564] T 00000000000000000000000000000000 P 26c9833d46fd47e1b35c1341716809ac [sys.catalog]: configured and running, proceeding with master startup.
I20260807 23:01:26.115921 23111 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26c9833d46fd47e1b35c1341716809ac [sys.catalog]: SysCatalogTable state changed. Reason: New leader 26c9833d46fd47e1b35c1341716809ac. Latest consensus state: current_term: 1 leader_uuid: "26c9833d46fd47e1b35c1341716809ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26c9833d46fd47e1b35c1341716809ac" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 42617 } } }
I20260807 23:01:26.116475 23111 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26c9833d46fd47e1b35c1341716809ac [sys.catalog]: This master's current role is: LEADER
I20260807 23:01:26.116304 23110 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26c9833d46fd47e1b35c1341716809ac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "26c9833d46fd47e1b35c1341716809ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26c9833d46fd47e1b35c1341716809ac" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 42617 } } }
I20260807 23:01:26.116596 23110 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26c9833d46fd47e1b35c1341716809ac [sys.catalog]: This master's current role is: LEADER
I20260807 23:01:26.119743 23117 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 23:01:26.125573 23117 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 23:01:26.136933 23117 catalog_manager.cc:1347] Generated new cluster ID: 4eb9f1001d6b4dd39f45ac63c0d114bc
I20260807 23:01:26.137089 23117 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 23:01:26.155454 23117 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 23:01:26.157084 23117 catalog_manager.cc:1504] Loading token signing keys...
I20260807 23:01:26.166631 23117 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 26c9833d46fd47e1b35c1341716809ac: Generated new TSK 0
I20260807 23:01:26.167578 23117 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 23:01:26.184348 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786143361779964-25976-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786143361779964-25976-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786143361779964-25976-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786143361779964-25976-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.1:0
--local_ip_for_outbound_sockets=127.25.94.1
--webserver_interface=127.25.94.1
--webserver_port=0
--tserver_master_addrs=127.25.94.62:42617
--builtin_ntp_servers=127.25.94.20:34677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260807 23:01:26.544621 23138 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 23:01:26.544984 23138 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 23:01:26.545153 23138 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 23:01:26.552754 23138 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 23:01:26.552899 23138 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 23:01:26.553076 23138 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.1
I20260807 23:01:26.562502 23138 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:34677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786143361779964-25976-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786143361779964-25976-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786143361779964-25976-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.94.1
--webserver_port=0
--tserver_master_addrs=127.25.94.62:42617
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.1
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786143361779964-25976-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 23:01:26.563881 23138 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 23:01:26.565717 23138 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 23:01:26.578657 23145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 23:01:26.578629 23147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 23:01:26.578751 23143 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 23:01:26.579336 23138 server_base.cc:1034] running on GCE node
I20260807 23:01:26.579958 23138 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 23:01:26.580884 23138 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 23:01:26.582093 23138 hybrid_clock.cc:648] HybridClock initialized: now 1786143686582053 us; error 31 us; skew 500 ppm
I20260807 23:01:26.582450 23138 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 23:01:26.584793 23138 webserver.cc:460] Webserver started at http://127.25.94.1:39609/ using document root <none> and password file <none>
I20260807 23:01:26.585564 23138 fs_manager.cc:359] Metadata directory not provided
I20260807 23:01:26.585681 23138 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 23:01:26.586005 23138 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 23:01:26.588845 23138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786143361779964-25976-0/minicluster-data/ts-0/data/instance:
uuid: "1fe87973db4149b387542b989c096f35"
format_stamp: "Formatted at 2026-08-07 23:01:26 on dist-test-slave-z8x9"
server_key: "d29954c11c9647e6c50d1bef783cfafa"
server_key_iv: "8a458ed66710e31e405beb3b3ad3a659"
server_key_version: "encryptionkey@0"
I20260807 23:01:26.589654 23138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786143361779964-25976-0/minicluster-data/ts-0/wal/instance:
uuid: "1fe87973db4149b387542b989c096f35"
format_stamp: "Formatted at 2026-08-07 23:01:26 on dist-test-slave-z8x9"
server_key: "d29954c11c9647e6c50d1bef783cfafa"
server_key_iv: "8a458ed66710e31e405beb3b3ad3a659"
server_key_version: "encryptionkey@0"
I20260807 23:01:26.594910 23138 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260807 23:01:26.598471 23161 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 23:01:26.599798 23138 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 23:01:26.599978 23138 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786143361779964-25976-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786143361779964-25976-0/minicluster-data/ts-0/wal
uuid: "1fe87973db4149b387542b989c096f35"
format_stamp: "Formatted at 2026-08-07 23:01:26 on dist-test-slave-z8x9"
server_key: "d29954c11c9647e6c50d1bef783cfafa"
server_key_iv: "8a458ed66710e31e405beb3b3ad3a659"
server_key_version: "encryptionkey@0"
I20260807 23:01:26.600162 23138 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786143361779964-25976-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786143361779964-25976-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786143361779964-25976-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 23:01:26.624110 23138 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 23:01:26.624960 23138 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 23:01:26.625308 23138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 23:01:26.626304 23138 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 23:01:26.627838 23138 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 23:01:26.627960 23138 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 23:01:26.628091 23138 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 23:01:26.628189 23138 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 23:01:26.671943 23355 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.1:46341 every 8 connection(s)
I20260807 23:01:26.672207 23138 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.1:46341
I20260807 23:01:26.674067 23138 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786143361779964-25976-0/minicluster-data/ts-0/data/info.pb
I20260807 23:01:26.683755 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 23138
I20260807 23:01:26.683957 25976 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786143361779964-25976-0/minicluster-data/ts-0/wal/instance
I20260807 23:01:26.684458 25976 external_mini_cluster.cc:1428] Setting key f8b37eeb36bc6dccef2731c55216d0d0
I20260807 23:01:26.687722 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786143361779964-25976-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786143361779964-25976-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786143361779964-25976-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786143361779964-25976-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.2:0
--local_ip_for_outbound_sockets=127.25.94.2
--webserver_interface=127.25.94.2
--webserver_port=0
--tserver_master_addrs=127.25.94.62:42617
--builtin_ntp_servers=127.25.94.20:34677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260807 23:01:26.687819 23360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:26.676897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.1:40871 (local address 127.25.94.62:42617)
0807 23:01:26.677544 (+ 647us) server_negotiation.cc:206] Beginning negotiation
0807 23:01:26.677550 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 23:01:26.678840 (+ 1290us) server_negotiation.cc:407] Connection header received
0807 23:01:26.680736 (+ 1896us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:01:26.680742 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:01:26.680840 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:01:26.680965 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:01:26.682140 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:26.682872 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:26.684622 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:26.685074 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:26.685232 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:26.686844 (+ 1612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:01:26.686851 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:01:26.686856 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 23:01:26.686877 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:01:26.686929 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 23:01:26.686946 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:01:26.687044 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0807 23:01:26.687269 (+ 225us) server_negotiation.cc:299] Negotiation successful
0807 23:01:26.687377 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":514,"thread_start_us":372,"threads_started":1}
I20260807 23:01:26.688637 23361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:26.677599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:42617 (local address 127.25.94.1:40871)
0807 23:01:26.678643 (+ 1044us) negotiation.cc:107] Waiting for socket to connect
0807 23:01:26.678693 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0807 23:01:26.680438 (+ 1745us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 23:01:26.681292 (+ 854us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 23:01:26.681301 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 23:01:26.681390 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0807 23:01:26.681968 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:01:26.681984 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:01:26.683050 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:01:26.683055 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:01:26.684434 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:01:26.684447 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:01:26.685420 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:01:26.685428 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:01:26.685702 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:26.686524 (+ 822us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 23:01:26.686527 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0807 23:01:26.686564 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 23:01:26.686648 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 23:01:26.687056 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 23:01:26.687065 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 23:01:26.687070 (+ 5us) client_negotiation.cc:769] Sending connection context
0807 23:01:26.687200 (+ 130us) client_negotiation.cc:240] Negotiation successful
0807 23:01:26.687513 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":344,"threads_started":1}
I20260807 23:01:26.690094 23357 heartbeater.cc:347] Connected to a master server at 127.25.94.62:42617
I20260807 23:01:26.690436 23357 heartbeater.cc:464] Registering TS with master...
I20260807 23:01:26.691339 23357 heartbeater.cc:511] Master 127.25.94.62:42617 requested a full tablet report, sending...
I20260807 23:01:26.693549 23038 ts_manager.cc:194] Registered new tserver with Master: 1fe87973db4149b387542b989c096f35 (127.25.94.1:46341)
I20260807 23:01:26.694945 23038 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.94.1:40871
W20260807 23:01:27.029312 23362 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 23:01:27.029597 23362 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 23:01:27.029744 23362 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 23:01:27.037051 23362 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 23:01:27.037250 23362 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 23:01:27.037448 23362 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.2
I20260807 23:01:27.047335 23362 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:34677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786143361779964-25976-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786143361779964-25976-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786143361779964-25976-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.94.2
--webserver_port=0
--tserver_master_addrs=127.25.94.62:42617
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.2
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786143361779964-25976-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 23:01:27.048758 23362 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 23:01:27.050731 23362 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 23:01:27.064137 23372 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 23:01:27.064096 23374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 23:01:27.064059 23376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 23:01:27.064904 23362 server_base.cc:1034] running on GCE node
I20260807 23:01:27.065593 23362 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 23:01:27.066570 23362 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 23:01:27.067772 23362 hybrid_clock.cc:648] HybridClock initialized: now 1786143687067734 us; error 40 us; skew 500 ppm
I20260807 23:01:27.068096 23362 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 23:01:27.070556 23362 webserver.cc:460] Webserver started at http://127.25.94.2:46401/ using document root <none> and password file <none>
I20260807 23:01:27.071228 23362 fs_manager.cc:359] Metadata directory not provided
I20260807 23:01:27.071326 23362 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 23:01:27.071663 23362 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 23:01:27.074558 23362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786143361779964-25976-0/minicluster-data/ts-1/data/instance:
uuid: "70561e00ea154a38ad150af53ada25cc"
format_stamp: "Formatted at 2026-08-07 23:01:27 on dist-test-slave-z8x9"
server_key: "920178fa1e423572813e32b479d2090b"
server_key_iv: "ff4847eb271fd306c8a58060f9d651b8"
server_key_version: "encryptionkey@0"
I20260807 23:01:27.075376 23362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786143361779964-25976-0/minicluster-data/ts-1/wal/instance:
uuid: "70561e00ea154a38ad150af53ada25cc"
format_stamp: "Formatted at 2026-08-07 23:01:27 on dist-test-slave-z8x9"
server_key: "920178fa1e423572813e32b479d2090b"
server_key_iv: "ff4847eb271fd306c8a58060f9d651b8"
server_key_version: "encryptionkey@0"
I20260807 23:01:27.080745 23362 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260807 23:01:27.084540 23387 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 23:01:27.086057 23362 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 23:01:27.086246 23362 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786143361779964-25976-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786143361779964-25976-0/minicluster-data/ts-1/wal
uuid: "70561e00ea154a38ad150af53ada25cc"
format_stamp: "Formatted at 2026-08-07 23:01:27 on dist-test-slave-z8x9"
server_key: "920178fa1e423572813e32b479d2090b"
server_key_iv: "ff4847eb271fd306c8a58060f9d651b8"
server_key_version: "encryptionkey@0"
I20260807 23:01:27.086481 23362 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786143361779964-25976-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786143361779964-25976-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786143361779964-25976-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 23:01:27.103739 23362 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 23:01:27.104693 23362 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 23:01:27.104986 23362 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 23:01:27.106047 23362 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 23:01:27.107573 23362 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 23:01:27.107704 23362 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 23:01:27.107847 23362 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 23:01:27.107939 23362 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 23:01:27.144474 23362 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.2:43823
I20260807 23:01:27.144533 23577 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.2:43823 every 8 connection(s)
I20260807 23:01:27.145936 23362 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786143361779964-25976-0/minicluster-data/ts-1/data/info.pb
I20260807 23:01:27.149863 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 23362
I20260807 23:01:27.150084 25976 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786143361779964-25976-0/minicluster-data/ts-1/wal/instance
I20260807 23:01:27.150583 25976 external_mini_cluster.cc:1428] Setting key b82b52d034681f58ab14189e53f82321
I20260807 23:01:27.153697 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786143361779964-25976-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786143361779964-25976-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786143361779964-25976-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786143361779964-25976-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.3:0
--local_ip_for_outbound_sockets=127.25.94.3
--webserver_interface=127.25.94.3
--webserver_port=0
--tserver_master_addrs=127.25.94.62:42617
--builtin_ntp_servers=127.25.94.20:34677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260807 23:01:27.159819 23360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:27.148851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.2:57395 (local address 127.25.94.62:42617)
0807 23:01:27.149077 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0807 23:01:27.149084 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 23:01:27.150798 (+ 1714us) server_negotiation.cc:407] Connection header received
0807 23:01:27.152926 (+ 2128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:01:27.152932 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:01:27.153022 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:01:27.153145 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:01:27.154412 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:27.155174 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:27.156795 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:27.157310 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:27.157447 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:27.158944 (+ 1497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:01:27.158950 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:01:27.158953 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 23:01:27.158973 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:01:27.159005 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 23:01:27.159017 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:01:27.159120 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0807 23:01:27.159366 (+ 246us) server_negotiation.cc:299] Negotiation successful
0807 23:01:27.159472 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260807 23:01:27.160879 23583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:27.149474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:42617 (local address 127.25.94.2:57395)
0807 23:01:27.150599 (+ 1125us) negotiation.cc:107] Waiting for socket to connect
0807 23:01:27.150660 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0807 23:01:27.152613 (+ 1953us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 23:01:27.153444 (+ 831us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 23:01:27.153455 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 23:01:27.153550 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0807 23:01:27.154234 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:01:27.154252 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:01:27.155334 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:01:27.155342 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:01:27.156637 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:01:27.156649 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:01:27.157560 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:01:27.157569 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:01:27.157803 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:27.158574 (+ 771us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 23:01:27.158576 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0807 23:01:27.158618 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 23:01:27.158703 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 23:01:27.159122 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 23:01:27.159136 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 23:01:27.159146 (+ 10us) client_negotiation.cc:769] Sending connection context
0807 23:01:27.159354 (+ 208us) client_negotiation.cc:240] Negotiation successful
0807 23:01:27.159777 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":372,"threads_started":1}
I20260807 23:01:27.162544 23578 heartbeater.cc:347] Connected to a master server at 127.25.94.62:42617
I20260807 23:01:27.162890 23578 heartbeater.cc:464] Registering TS with master...
I20260807 23:01:27.163986 23578 heartbeater.cc:511] Master 127.25.94.62:42617 requested a full tablet report, sending...
I20260807 23:01:27.165856 23038 ts_manager.cc:194] Registered new tserver with Master: 70561e00ea154a38ad150af53ada25cc (127.25.94.2:43823)
I20260807 23:01:27.166819 23038 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.94.2:57395
W20260807 23:01:27.513321 23584 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 23:01:27.513693 23584 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 23:01:27.513911 23584 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 23:01:27.521651 23584 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 23:01:27.521761 23584 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 23:01:27.521872 23584 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.3
I20260807 23:01:27.531513 23584 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:34677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786143361779964-25976-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786143361779964-25976-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786143361779964-25976-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.94.3
--webserver_port=0
--tserver_master_addrs=127.25.94.62:42617
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.3
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786143361779964-25976-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 23:01:27.532814 23584 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 23:01:27.534724 23584 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 23:01:27.548658 23592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 23:01:27.548614 23591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 23:01:27.548898 23594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 23:01:27.549705 23584 server_base.cc:1034] running on GCE node
I20260807 23:01:27.550362 23584 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 23:01:27.551402 23584 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 23:01:27.552714 23584 hybrid_clock.cc:648] HybridClock initialized: now 1786143687552665 us; error 74 us; skew 500 ppm
I20260807 23:01:27.553071 23584 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 23:01:27.555696 23584 webserver.cc:460] Webserver started at http://127.25.94.3:37089/ using document root <none> and password file <none>
I20260807 23:01:27.556469 23584 fs_manager.cc:359] Metadata directory not provided
I20260807 23:01:27.556610 23584 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 23:01:27.556945 23584 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 23:01:27.559935 23584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786143361779964-25976-0/minicluster-data/ts-2/data/instance:
uuid: "de834fd21a92405981281fa51418d325"
format_stamp: "Formatted at 2026-08-07 23:01:27 on dist-test-slave-z8x9"
server_key: "5f05bcc24d534d58c7d176dec9cee831"
server_key_iv: "3d0471be3d8c450410cb307f5e14187d"
server_key_version: "encryptionkey@0"
I20260807 23:01:27.560739 23584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786143361779964-25976-0/minicluster-data/ts-2/wal/instance:
uuid: "de834fd21a92405981281fa51418d325"
format_stamp: "Formatted at 2026-08-07 23:01:27 on dist-test-slave-z8x9"
server_key: "5f05bcc24d534d58c7d176dec9cee831"
server_key_iv: "3d0471be3d8c450410cb307f5e14187d"
server_key_version: "encryptionkey@0"
I20260807 23:01:27.566058 23584 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260807 23:01:27.569514 23604 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 23:01:27.570784 23584 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260807 23:01:27.570968 23584 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786143361779964-25976-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786143361779964-25976-0/minicluster-data/ts-2/wal
uuid: "de834fd21a92405981281fa51418d325"
format_stamp: "Formatted at 2026-08-07 23:01:27 on dist-test-slave-z8x9"
server_key: "5f05bcc24d534d58c7d176dec9cee831"
server_key_iv: "3d0471be3d8c450410cb307f5e14187d"
server_key_version: "encryptionkey@0"
I20260807 23:01:27.571219 23584 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786143361779964-25976-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786143361779964-25976-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786143361779964-25976-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 23:01:27.602941 23584 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 23:01:27.603916 23584 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 23:01:27.604213 23584 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 23:01:27.605432 23584 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 23:01:27.607055 23584 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 23:01:27.607199 23584 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 23:01:27.607358 23584 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 23:01:27.607452 23584 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 23:01:27.643726 23584 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.3:45667
I20260807 23:01:27.643841 23778 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.3:45667 every 8 connection(s)
I20260807 23:01:27.645097 23584 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786143361779964-25976-0/minicluster-data/ts-2/data/info.pb
I20260807 23:01:27.646710 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 23584
I20260807 23:01:27.646921 25976 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786143361779964-25976-0/minicluster-data/ts-2/wal/instance
I20260807 23:01:27.647509 25976 external_mini_cluster.cc:1428] Setting key 752f96e867796772edfb5cf4e3e4c21b
I20260807 23:01:27.658541 23360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:27.647984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.3:39433 (local address 127.25.94.62:42617)
0807 23:01:27.648288 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0807 23:01:27.648295 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 23:01:27.649962 (+ 1667us) server_negotiation.cc:407] Connection header received
0807 23:01:27.651817 (+ 1855us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:01:27.651822 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:01:27.651895 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:01:27.651988 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:01:27.653068 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:27.653893 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:27.655753 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:27.656158 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:27.656271 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:27.657840 (+ 1569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:01:27.657845 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:01:27.657848 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 23:01:27.657872 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:01:27.657904 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 23:01:27.657916 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:01:27.657989 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0807 23:01:27.658186 (+ 197us) server_negotiation.cc:299] Negotiation successful
0807 23:01:27.658263 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260807 23:01:27.659757 23783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:27.648609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:42617 (local address 127.25.94.3:39433)
0807 23:01:27.649791 (+ 1182us) negotiation.cc:107] Waiting for socket to connect
0807 23:01:27.649840 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0807 23:01:27.651533 (+ 1693us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 23:01:27.652257 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 23:01:27.652266 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 23:01:27.652355 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0807 23:01:27.652924 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:01:27.652940 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:01:27.654091 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:01:27.654096 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:01:27.655610 (+ 1514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:01:27.655622 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:01:27.656399 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:01:27.656406 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:01:27.656632 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:27.657499 (+ 867us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 23:01:27.657503 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0807 23:01:27.657560 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 23:01:27.657662 (+ 102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 23:01:27.658003 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 23:01:27.658010 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 23:01:27.658016 (+ 6us) client_negotiation.cc:769] Sending connection context
0807 23:01:27.658158 (+ 142us) client_negotiation.cc:240] Negotiation successful
0807 23:01:27.658455 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":618,"thread_start_us":380,"threads_started":1}
I20260807 23:01:27.661342 23780 heartbeater.cc:347] Connected to a master server at 127.25.94.62:42617
I20260807 23:01:27.661666 23780 heartbeater.cc:464] Registering TS with master...
I20260807 23:01:27.662672 23780 heartbeater.cc:511] Master 127.25.94.62:42617 requested a full tablet report, sending...
I20260807 23:01:27.664227 23038 ts_manager.cc:194] Registered new tserver with Master: de834fd21a92405981281fa51418d325 (127.25.94.3:45667)
I20260807 23:01:27.665354 23038 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.94.3:39433
I20260807 23:01:27.676884 25976 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 23:01:27.687898 23360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:27.681241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52008 (local address 127.25.94.62:42617)
0807 23:01:27.681412 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0807 23:01:27.681417 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 23:01:27.681891 (+ 474us) server_negotiation.cc:407] Connection header received
0807 23:01:27.682040 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:01:27.682044 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:01:27.682127 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:01:27.682241 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:01:27.683100 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:27.683849 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:27.685551 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:27.686003 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:27.686112 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:27.687276 (+ 1164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:01:27.687282 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:01:27.687285 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 23:01:27.687305 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:01:27.687337 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 23:01:27.687349 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:01:27.687423 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0807 23:01:27.687567 (+ 144us) server_negotiation.cc:299] Negotiation successful
0807 23:01:27.687647 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260807 23:01:27.692055 23038 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52008:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260807 23:01:27.698529 23357 heartbeater.cc:503] Master 127.25.94.62:42617 was elected leader, sending a full tablet report...
W20260807 23:01:27.700114 23038 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 23:01:27.723367 23794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:27.711918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.3:45667 (local address 127.0.0.1:53076)
0807 23:01:27.712752 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0807 23:01:27.712793 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0807 23:01:27.712969 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 23:01:27.714864 (+ 1895us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 23:01:27.714873 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 23:01:27.714907 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 23:01:27.715706 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:01:27.715725 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:01:27.717759 (+ 2034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:01:27.717764 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:01:27.720313 (+ 2549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:01:27.720333 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:01:27.722283 (+ 1950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:01:27.722290 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:01:27.722497 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:27.722563 (+ 66us) client_negotiation.cc:769] Sending connection context
0807 23:01:27.722729 (+ 166us) client_negotiation.cc:240] Negotiation successful
0807 23:01:27.722931 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"mutex_wait_us":256,"spinlock_wait_cycles":12032,"thread_start_us":919,"threads_started":1}
I20260807 23:01:27.723336 23798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:27.712383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53076 (local address 127.25.94.3:45667)
0807 23:01:27.714146 (+ 1763us) server_negotiation.cc:206] Beginning negotiation
0807 23:01:27.714154 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 23:01:27.714186 (+ 32us) server_negotiation.cc:407] Connection header received
0807 23:01:27.714337 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:01:27.714349 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:01:27.714627 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:01:27.715255 (+ 628us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 23:01:27.715955 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:27.717583 (+ 1628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:27.720577 (+ 2994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:27.722050 (+ 1473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:27.722235 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:27.722532 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0807 23:01:27.722772 (+ 240us) server_negotiation.cc:299] Negotiation successful
0807 23:01:27.722941 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1586,"thread_start_us":1432,"threads_started":1}
I20260807 23:01:27.723613 23799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:27.712659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.2:43823 (local address 127.0.0.1:54780)
0807 23:01:27.714012 (+ 1353us) negotiation.cc:107] Waiting for socket to connect
0807 23:01:27.714061 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0807 23:01:27.714184 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 23:01:27.715078 (+ 894us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 23:01:27.715082 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 23:01:27.715148 (+ 66us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 23:01:27.715735 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:01:27.715758 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:01:27.717558 (+ 1800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:01:27.717567 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:01:27.720444 (+ 2877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:01:27.720459 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:01:27.722709 (+ 2250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:01:27.722717 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:01:27.722927 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:27.722960 (+ 33us) client_negotiation.cc:769] Sending connection context
0807 23:01:27.723051 (+ 91us) client_negotiation.cc:240] Negotiation successful
0807 23:01:27.723189 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1231,"mutex_wait_us":489,"thread_start_us":1597,"threads_started":1}
I20260807 23:01:27.724404 23800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:27.712897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54780 (local address 127.25.94.2:43823)
0807 23:01:27.713637 (+ 740us) server_negotiation.cc:206] Beginning negotiation
0807 23:01:27.713649 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 23:01:27.714182 (+ 533us) server_negotiation.cc:407] Connection header received
0807 23:01:27.714331 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:01:27.714339 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:01:27.714630 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:01:27.714835 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 23:01:27.715968 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:27.717428 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:27.720634 (+ 3206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:27.722103 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:27.722273 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:27.722532 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0807 23:01:27.723879 (+ 1347us) server_negotiation.cc:299] Negotiation successful
0807 23:01:27.724046 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":582,"thread_start_us":476,"threads_started":1}
I20260807 23:01:27.725567 23801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:27.713729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.1:46341 (local address 127.0.0.1:36062)
0807 23:01:27.714845 (+ 1116us) negotiation.cc:107] Waiting for socket to connect
0807 23:01:27.714862 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 23:01:27.715007 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 23:01:27.716706 (+ 1699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 23:01:27.716711 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 23:01:27.716739 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 23:01:27.717062 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:01:27.717071 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:01:27.719250 (+ 2179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:01:27.719254 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:01:27.721202 (+ 1948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:01:27.721218 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:01:27.724749 (+ 3531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:01:27.724757 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:01:27.724933 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:27.724965 (+ 32us) client_negotiation.cc:769] Sending connection context
0807 23:01:27.725049 (+ 84us) client_negotiation.cc:240] Negotiation successful
0807 23:01:27.725209 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":971,"mutex_wait_us":3,"spinlock_wait_cycles":384,"thread_start_us":840,"threads_started":1}
I20260807 23:01:27.726236 23802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:27.714348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36062 (local address 127.25.94.1:46341)
0807 23:01:27.716131 (+ 1783us) server_negotiation.cc:206] Beginning negotiation
0807 23:01:27.716138 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 23:01:27.716166 (+ 28us) server_negotiation.cc:407] Connection header received
0807 23:01:27.716263 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:01:27.716270 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:01:27.716487 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:01:27.716686 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 23:01:27.717356 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:27.719046 (+ 1690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:27.721413 (+ 2367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:27.724034 (+ 2621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:27.724174 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:27.724450 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0807 23:01:27.725363 (+ 913us) server_negotiation.cc:299] Negotiation successful
0807 23:01:27.725651 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1596,"thread_start_us":1472,"threads_started":1}
I20260807 23:01:27.728955 23676 tablet_service.cc:1467] Processing CreateTablet for tablet 95096e4292e644e5ba5c6bee1d6ff10c (DEFAULT_TABLE table=test-table [id=ecdf6fdb29334b209a5f7086519e8ca5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 23:01:27.730327 23676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95096e4292e644e5ba5c6bee1d6ff10c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 23:01:27.730243 23466 tablet_service.cc:1467] Processing CreateTablet for tablet 95096e4292e644e5ba5c6bee1d6ff10c (DEFAULT_TABLE table=test-table [id=ecdf6fdb29334b209a5f7086519e8ca5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 23:01:27.730346 23242 tablet_service.cc:1467] Processing CreateTablet for tablet 95096e4292e644e5ba5c6bee1d6ff10c (DEFAULT_TABLE table=test-table [id=ecdf6fdb29334b209a5f7086519e8ca5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 23:01:27.731583 23466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95096e4292e644e5ba5c6bee1d6ff10c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 23:01:27.731588 23242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95096e4292e644e5ba5c6bee1d6ff10c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 23:01:27.742280 23805 tablet_bootstrap.cc:492] T 95096e4292e644e5ba5c6bee1d6ff10c P de834fd21a92405981281fa51418d325: Bootstrap starting.
I20260807 23:01:27.746220 23805 tablet_bootstrap.cc:654] T 95096e4292e644e5ba5c6bee1d6ff10c P de834fd21a92405981281fa51418d325: Neither blocks nor log segments found. Creating new log.
I20260807 23:01:27.747352 23810 tablet_bootstrap.cc:492] T 95096e4292e644e5ba5c6bee1d6ff10c P 70561e00ea154a38ad150af53ada25cc: Bootstrap starting.
I20260807 23:01:27.747614 23805 log.cc:824] T 95096e4292e644e5ba5c6bee1d6ff10c P de834fd21a92405981281fa51418d325: Log is configured to *not* fsync() on all Append() calls
I20260807 23:01:27.749212 23809 tablet_bootstrap.cc:492] T 95096e4292e644e5ba5c6bee1d6ff10c P 1fe87973db4149b387542b989c096f35: Bootstrap starting.
I20260807 23:01:27.752269 23809 tablet_bootstrap.cc:654] T 95096e4292e644e5ba5c6bee1d6ff10c P 1fe87973db4149b387542b989c096f35: Neither blocks nor log segments found. Creating new log.
I20260807 23:01:27.752363 23810 tablet_bootstrap.cc:654] T 95096e4292e644e5ba5c6bee1d6ff10c P 70561e00ea154a38ad150af53ada25cc: Neither blocks nor log segments found. Creating new log.
I20260807 23:01:27.753511 23809 log.cc:824] T 95096e4292e644e5ba5c6bee1d6ff10c P 1fe87973db4149b387542b989c096f35: Log is configured to *not* fsync() on all Append() calls
I20260807 23:01:27.753907 23810 log.cc:824] T 95096e4292e644e5ba5c6bee1d6ff10c P 70561e00ea154a38ad150af53ada25cc: Log is configured to *not* fsync() on all Append() calls
I20260807 23:01:27.757869 23809 tablet_bootstrap.cc:492] T 95096e4292e644e5ba5c6bee1d6ff10c P 1fe87973db4149b387542b989c096f35: No bootstrap required, opened a new log
I20260807 23:01:27.757872 23805 tablet_bootstrap.cc:492] T 95096e4292e644e5ba5c6bee1d6ff10c P de834fd21a92405981281fa51418d325: No bootstrap required, opened a new log
I20260807 23:01:27.758170 23805 ts_tablet_manager.cc:1397] T 95096e4292e644e5ba5c6bee1d6ff10c P de834fd21a92405981281fa51418d325: Time spent bootstrapping tablet: real 0.016s user 0.010s sys 0.004s
I20260807 23:01:27.758170 23809 ts_tablet_manager.cc:1397] T 95096e4292e644e5ba5c6bee1d6ff10c P 1fe87973db4149b387542b989c096f35: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260807 23:01:27.758405 23810 tablet_bootstrap.cc:492] T 95096e4292e644e5ba5c6bee1d6ff10c P 70561e00ea154a38ad150af53ada25cc: No bootstrap required, opened a new log
I20260807 23:01:27.758675 23810 ts_tablet_manager.cc:1397] T 95096e4292e644e5ba5c6bee1d6ff10c P 70561e00ea154a38ad150af53ada25cc: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.005s
I20260807 23:01:27.765990 23810 raft_consensus.cc:348] T 95096e4292e644e5ba5c6bee1d6ff10c P 70561e00ea154a38ad150af53ada25cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de834fd21a92405981281fa51418d325" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 45667 } } peers { permanent_uuid: "1fe87973db4149b387542b989c096f35" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 46341 } } peers { permanent_uuid: "70561e00ea154a38ad150af53ada25cc" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 43823 } }
I20260807 23:01:27.766394 23810 raft_consensus.cc:374] T 95096e4292e644e5ba5c6bee1d6ff10c P 70561e00ea154a38ad150af53ada25cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 23:01:27.766490 23810 raft_consensus.cc:729] T 95096e4292e644e5ba5c6bee1d6ff10c P 70561e00ea154a38ad150af53ada25cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70561e00ea154a38ad150af53ada25cc, State: Initialized, Role: FOLLOWER
I20260807 23:01:27.767143 23810 consensus_queue.cc:260] T 95096e4292e644e5ba5c6bee1d6ff10c P 70561e00ea154a38ad150af53ada25cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de834fd21a92405981281fa51418d325" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 45667 } } peers { permanent_uuid: "1fe87973db4149b387542b989c096f35" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 46341 } } peers { permanent_uuid: "70561e00ea154a38ad150af53ada25cc" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 43823 } }
I20260807 23:01:27.768272 23578 heartbeater.cc:503] Master 127.25.94.62:42617 was elected leader, sending a full tablet report...
I20260807 23:01:27.768985 23810 ts_tablet_manager.cc:1428] T 95096e4292e644e5ba5c6bee1d6ff10c P 70561e00ea154a38ad150af53ada25cc: Time spent starting tablet: real 0.010s user 0.011s sys 0.000s
I20260807 23:01:27.769112 23809 raft_consensus.cc:348] T 95096e4292e644e5ba5c6bee1d6ff10c P 1fe87973db4149b387542b989c096f35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de834fd21a92405981281fa51418d325" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 45667 } } peers { permanent_uuid: "1fe87973db4149b387542b989c096f35" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 46341 } } peers { permanent_uuid: "70561e00ea154a38ad150af53ada25cc" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 43823 } }
I20260807 23:01:27.769107 23805 raft_consensus.cc:348] T 95096e4292e644e5ba5c6bee1d6ff10c P de834fd21a92405981281fa51418d325 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de834fd21a92405981281fa51418d325" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 45667 } } peers { permanent_uuid: "1fe87973db4149b387542b989c096f35" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 46341 } } peers { permanent_uuid: "70561e00ea154a38ad150af53ada25cc" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 43823 } }
I20260807 23:01:27.769572 23809 raft_consensus.cc:374] T 95096e4292e644e5ba5c6bee1d6ff10c P 1fe87973db4149b387542b989c096f35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 23:01:27.769572 23805 raft_consensus.cc:374] T 95096e4292e644e5ba5c6bee1d6ff10c P de834fd21a92405981281fa51418d325 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 23:01:27.769690 23805 raft_consensus.cc:729] T 95096e4292e644e5ba5c6bee1d6ff10c P de834fd21a92405981281fa51418d325 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de834fd21a92405981281fa51418d325, State: Initialized, Role: FOLLOWER
I20260807 23:01:27.769690 23809 raft_consensus.cc:729] T 95096e4292e644e5ba5c6bee1d6ff10c P 1fe87973db4149b387542b989c096f35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fe87973db4149b387542b989c096f35, State: Initialized, Role: FOLLOWER
I20260807 23:01:27.770314 23809 consensus_queue.cc:260] T 95096e4292e644e5ba5c6bee1d6ff10c P 1fe87973db4149b387542b989c096f35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de834fd21a92405981281fa51418d325" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 45667 } } peers { permanent_uuid: "1fe87973db4149b387542b989c096f35" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 46341 } } peers { permanent_uuid: "70561e00ea154a38ad150af53ada25cc" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 43823 } }
I20260807 23:01:27.770319 23805 consensus_queue.cc:260] T 95096e4292e644e5ba5c6bee1d6ff10c P de834fd21a92405981281fa51418d325 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de834fd21a92405981281fa51418d325" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 45667 } } peers { permanent_uuid: "1fe87973db4149b387542b989c096f35" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 46341 } } peers { permanent_uuid: "70561e00ea154a38ad150af53ada25cc" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 43823 } }
I20260807 23:01:27.772300 23809 ts_tablet_manager.cc:1428] T 95096e4292e644e5ba5c6bee1d6ff10c P 1fe87973db4149b387542b989c096f35: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260807 23:01:27.773988 23780 heartbeater.cc:503] Master 127.25.94.62:42617 was elected leader, sending a full tablet report...
I20260807 23:01:27.775103 23805 ts_tablet_manager.cc:1428] T 95096e4292e644e5ba5c6bee1d6ff10c P de834fd21a92405981281fa51418d325: Time spent starting tablet: real 0.016s user 0.010s sys 0.006s
W20260807 23:01:27.896981 23781 tablet.cc:2369] T 95096e4292e644e5ba5c6bee1d6ff10c P de834fd21a92405981281fa51418d325: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 23:01:27.898100 23580 tablet.cc:2369] T 95096e4292e644e5ba5c6bee1d6ff10c P 70561e00ea154a38ad150af53ada25cc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 23:01:27.926985 23358 tablet.cc:2369] T 95096e4292e644e5ba5c6bee1d6ff10c P 1fe87973db4149b387542b989c096f35: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 23:01:27.994505 23820 raft_consensus.cc:482] T 95096e4292e644e5ba5c6bee1d6ff10c P de834fd21a92405981281fa51418d325 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 23:01:27.994876 23820 raft_consensus.cc:504] T 95096e4292e644e5ba5c6bee1d6ff10c P de834fd21a92405981281fa51418d325 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de834fd21a92405981281fa51418d325" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 45667 } } peers { permanent_uuid: "1fe87973db4149b387542b989c096f35" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 46341 } } peers { permanent_uuid: "70561e00ea154a38ad150af53ada25cc" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 43823 } }
I20260807 23:01:27.996657 23820 leader_election.cc:288] T 95096e4292e644e5ba5c6bee1d6ff10c P de834fd21a92405981281fa51418d325 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1fe87973db4149b387542b989c096f35 (127.25.94.1:46341), 70561e00ea154a38ad150af53ada25cc (127.25.94.2:43823)
I20260807 23:01:28.003468 23783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:27.996958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.2:43823 (local address 127.25.94.3:35709)
0807 23:01:27.997403 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0807 23:01:27.997430 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0807 23:01:27.997559 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 23:01:27.997993 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 23:01:27.997998 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 23:01:27.998026 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 23:01:27.998256 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:01:27.998269 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:01:27.999574 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:01:27.999580 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:01:28.001295 (+ 1715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:01:28.001310 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:01:28.002688 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:01:28.002694 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:01:28.002885 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:28.002924 (+ 39us) client_negotiation.cc:769] Sending connection context
0807 23:01:28.003016 (+ 92us) client_negotiation.cc:240] Negotiation successful
0807 23:01:28.003157 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302}
I20260807 23:01:28.003495 23800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:27.997316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.3:35709 (local address 127.25.94.2:43823)
0807 23:01:27.997526 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0807 23:01:27.997531 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 23:01:27.997551 (+ 20us) server_negotiation.cc:407] Connection header received
0807 23:01:27.997737 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:01:27.997743 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:01:27.997811 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:01:27.997913 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 23:01:27.998424 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:27.999429 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:28.001457 (+ 2028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:28.002532 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:28.002629 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:28.002697 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0807 23:01:28.003061 (+ 364us) server_negotiation.cc:299] Negotiation successful
0807 23:01:28.003158 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260807 23:01:28.004310 23821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:27.997101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.1:46341 (local address 127.25.94.3:55695)
0807 23:01:27.998052 (+ 951us) negotiation.cc:107] Waiting for socket to connect
0807 23:01:27.998077 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0807 23:01:27.998192 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 23:01:27.998552 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 23:01:27.998557 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 23:01:27.998582 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 23:01:27.998971 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:01:27.998983 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:01:28.000272 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:01:28.000276 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:01:28.002037 (+ 1761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:01:28.002053 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:01:28.003514 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:01:28.003523 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:01:28.003718 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:28.003759 (+ 41us) client_negotiation.cc:769] Sending connection context
0807 23:01:28.003863 (+ 104us) client_negotiation.cc:240] Negotiation successful
0807 23:01:28.004007 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":846,"thread_start_us":486,"threads_started":1}
I20260807 23:01:28.004300 23802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:27.997190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.3:55695 (local address 127.25.94.1:46341)
0807 23:01:27.997414 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0807 23:01:27.997422 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 23:01:27.998175 (+ 753us) server_negotiation.cc:407] Connection header received
0807 23:01:27.998305 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:01:27.998312 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:01:27.998412 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:01:27.998501 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 23:01:27.999137 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:28.000096 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:28.002252 (+ 2156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:28.003315 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:28.003430 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:28.003538 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0807 23:01:28.003904 (+ 366us) server_negotiation.cc:299] Negotiation successful
0807 23:01:28.004031 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260807 23:01:28.004871 23503 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95096e4292e644e5ba5c6bee1d6ff10c" candidate_uuid: "de834fd21a92405981281fa51418d325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70561e00ea154a38ad150af53ada25cc" is_pre_election: true
I20260807 23:01:28.005514 23503 raft_consensus.cc:2454] T 95096e4292e644e5ba5c6bee1d6ff10c P 70561e00ea154a38ad150af53ada25cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de834fd21a92405981281fa51418d325 in term 0.
I20260807 23:01:28.005502 23279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95096e4292e644e5ba5c6bee1d6ff10c" candidate_uuid: "de834fd21a92405981281fa51418d325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fe87973db4149b387542b989c096f35" is_pre_election: true
I20260807 23:01:28.006057 23279 raft_consensus.cc:2454] T 95096e4292e644e5ba5c6bee1d6ff10c P 1fe87973db4149b387542b989c096f35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de834fd21a92405981281fa51418d325 in term 0.
I20260807 23:01:28.006186 23607 leader_election.cc:302] T 95096e4292e644e5ba5c6bee1d6ff10c P de834fd21a92405981281fa51418d325 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 70561e00ea154a38ad150af53ada25cc, de834fd21a92405981281fa51418d325; no voters:
I20260807 23:01:28.006655 23820 raft_consensus.cc:2781] T 95096e4292e644e5ba5c6bee1d6ff10c P de834fd21a92405981281fa51418d325 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 23:01:28.006820 23820 raft_consensus.cc:482] T 95096e4292e644e5ba5c6bee1d6ff10c P de834fd21a92405981281fa51418d325 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 23:01:28.006935 23820 raft_consensus.cc:3037] T 95096e4292e644e5ba5c6bee1d6ff10c P de834fd21a92405981281fa51418d325 [term 0 FOLLOWER]: Advancing to term 1
I20260807 23:01:28.009420 23820 raft_consensus.cc:504] T 95096e4292e644e5ba5c6bee1d6ff10c P de834fd21a92405981281fa51418d325 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de834fd21a92405981281fa51418d325" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 45667 } } peers { permanent_uuid: "1fe87973db4149b387542b989c096f35" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 46341 } } peers { permanent_uuid: "70561e00ea154a38ad150af53ada25cc" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 43823 } }
I20260807 23:01:28.009999 23820 leader_election.cc:288] T 95096e4292e644e5ba5c6bee1d6ff10c P de834fd21a92405981281fa51418d325 [CANDIDATE]: Term 1 election: Requested vote from peers 1fe87973db4149b387542b989c096f35 (127.25.94.1:46341), 70561e00ea154a38ad150af53ada25cc (127.25.94.2:43823)
I20260807 23:01:28.010479 23279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95096e4292e644e5ba5c6bee1d6ff10c" candidate_uuid: "de834fd21a92405981281fa51418d325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fe87973db4149b387542b989c096f35"
I20260807 23:01:28.010526 23503 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95096e4292e644e5ba5c6bee1d6ff10c" candidate_uuid: "de834fd21a92405981281fa51418d325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70561e00ea154a38ad150af53ada25cc"
I20260807 23:01:28.010720 23503 raft_consensus.cc:3037] T 95096e4292e644e5ba5c6bee1d6ff10c P 70561e00ea154a38ad150af53ada25cc [term 0 FOLLOWER]: Advancing to term 1
I20260807 23:01:28.010720 23279 raft_consensus.cc:3037] T 95096e4292e644e5ba5c6bee1d6ff10c P 1fe87973db4149b387542b989c096f35 [term 0 FOLLOWER]: Advancing to term 1
I20260807 23:01:28.014038 23503 raft_consensus.cc:2454] T 95096e4292e644e5ba5c6bee1d6ff10c P 70561e00ea154a38ad150af53ada25cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de834fd21a92405981281fa51418d325 in term 1.
I20260807 23:01:28.014038 23279 raft_consensus.cc:2454] T 95096e4292e644e5ba5c6bee1d6ff10c P 1fe87973db4149b387542b989c096f35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de834fd21a92405981281fa51418d325 in term 1.
I20260807 23:01:28.014497 23610 leader_election.cc:302] T 95096e4292e644e5ba5c6bee1d6ff10c P de834fd21a92405981281fa51418d325 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fe87973db4149b387542b989c096f35, de834fd21a92405981281fa51418d325; no voters:
I20260807 23:01:28.014808 23820 raft_consensus.cc:2781] T 95096e4292e644e5ba5c6bee1d6ff10c P de834fd21a92405981281fa51418d325 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 23:01:28.015278 23820 raft_consensus.cc:686] T 95096e4292e644e5ba5c6bee1d6ff10c P de834fd21a92405981281fa51418d325 [term 1 LEADER]: Becoming Leader. State: Replica: de834fd21a92405981281fa51418d325, State: Running, Role: LEADER
I20260807 23:01:28.015878 23820 consensus_queue.cc:237] T 95096e4292e644e5ba5c6bee1d6ff10c P de834fd21a92405981281fa51418d325 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de834fd21a92405981281fa51418d325" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 45667 } } peers { permanent_uuid: "1fe87973db4149b387542b989c096f35" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 46341 } } peers { permanent_uuid: "70561e00ea154a38ad150af53ada25cc" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 43823 } }
I20260807 23:01:28.020706 23038 catalog_manager.cc:5495] T 95096e4292e644e5ba5c6bee1d6ff10c P de834fd21a92405981281fa51418d325 reported cstate change: term changed from 0 to 1, leader changed from <none> to de834fd21a92405981281fa51418d325 (127.25.94.3). New cstate: current_term: 1 leader_uuid: "de834fd21a92405981281fa51418d325" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de834fd21a92405981281fa51418d325" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 45667 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1fe87973db4149b387542b989c096f35" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 46341 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "70561e00ea154a38ad150af53ada25cc" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 43823 } health_report { overall_health: UNKNOWN } } }
I20260807 23:01:28.093532 23798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:28.087676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53078 (local address 127.25.94.3:45667)
0807 23:01:28.087914 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0807 23:01:28.087920 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 23:01:28.087940 (+ 20us) server_negotiation.cc:407] Connection header received
0807 23:01:28.088053 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:01:28.088057 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:01:28.088151 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:01:28.088312 (+ 161us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 23:01:28.088674 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:28.089461 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:28.090987 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:28.091504 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:28.091604 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:28.092239 (+ 635us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 23:01:28.092845 (+ 606us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 23:01:28.092970 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0807 23:01:28.093116 (+ 146us) server_negotiation.cc:299] Negotiation successful
0807 23:01:28.093217 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260807 23:01:28.102056 23503 raft_consensus.cc:1261] T 95096e4292e644e5ba5c6bee1d6ff10c P 70561e00ea154a38ad150af53ada25cc [term 1 FOLLOWER]: Refusing update from remote peer de834fd21a92405981281fa51418d325: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 23:01:28.102062 23279 raft_consensus.cc:1261] T 95096e4292e644e5ba5c6bee1d6ff10c P 1fe87973db4149b387542b989c096f35 [term 1 FOLLOWER]: Refusing update from remote peer de834fd21a92405981281fa51418d325: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 23:01:28.103123 23824 consensus_queue.cc:1035] T 95096e4292e644e5ba5c6bee1d6ff10c P de834fd21a92405981281fa51418d325 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fe87973db4149b387542b989c096f35" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 46341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 23:01:28.103552 23820 consensus_queue.cc:1035] T 95096e4292e644e5ba5c6bee1d6ff10c P de834fd21a92405981281fa51418d325 [LEADER]: Connected to new peer: Peer: permanent_uuid: "70561e00ea154a38ad150af53ada25cc" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 43823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 23:01:28.115515 23827 mvcc.cc:205] Tried to move back new op lower bound from 7316044546455359488 to 7316044546117337088. Current Snapshot: MvccSnapshot[applied={T|T < 7316044546455359488}]
I20260807 23:01:28.139751 23037 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52008:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 23:01:28.140101 23037 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52008:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 23:01:28.144449 23037 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 26c9833d46fd47e1b35c1341716809ac: Sending DeleteTablet for 3 replicas of tablet 95096e4292e644e5ba5c6bee1d6ff10c
I20260807 23:01:28.145782 23242 tablet_service.cc:1514] Processing DeleteTablet for tablet 95096e4292e644e5ba5c6bee1d6ff10c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 23:01:28 UTC) from {username='slave'} at 127.0.0.1:36062
I20260807 23:01:28.148983 23676 tablet_service.cc:1514] Processing DeleteTablet for tablet 95096e4292e644e5ba5c6bee1d6ff10c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 23:01:28 UTC) from {username='slave'} at 127.0.0.1:53076
I20260807 23:01:28.149603 23466 tablet_service.cc:1514] Processing DeleteTablet for tablet 95096e4292e644e5ba5c6bee1d6ff10c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 23:01:28 UTC) from {username='slave'} at 127.0.0.1:54780
I20260807 23:01:28.152050 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 23138
I20260807 23:01:28.158316 23851 tablet_replica.cc:330] T 95096e4292e644e5ba5c6bee1d6ff10c P 70561e00ea154a38ad150af53ada25cc: stopping tablet replica
I20260807 23:01:28.158933 23851 raft_consensus.cc:2229] T 95096e4292e644e5ba5c6bee1d6ff10c P 70561e00ea154a38ad150af53ada25cc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 23:01:28.159603 23851 raft_consensus.cc:2258] T 95096e4292e644e5ba5c6bee1d6ff10c P 70561e00ea154a38ad150af53ada25cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 23:01:28.166715 23850 tablet_replica.cc:330] T 95096e4292e644e5ba5c6bee1d6ff10c P de834fd21a92405981281fa51418d325: stopping tablet replica
I20260807 23:01:28.167363 23850 raft_consensus.cc:2229] T 95096e4292e644e5ba5c6bee1d6ff10c P de834fd21a92405981281fa51418d325 [term 1 LEADER]: Raft consensus shutting down.
I20260807 23:01:28.167594 23849 tablet_replica.cc:330] T 95096e4292e644e5ba5c6bee1d6ff10c P 1fe87973db4149b387542b989c096f35: stopping tablet replica
I20260807 23:01:28.168119 23849 raft_consensus.cc:2229] T 95096e4292e644e5ba5c6bee1d6ff10c P 1fe87973db4149b387542b989c096f35 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 23:01:28.168334 23850 raft_consensus.cc:2258] T 95096e4292e644e5ba5c6bee1d6ff10c P de834fd21a92405981281fa51418d325 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 23:01:28.168833 23849 raft_consensus.cc:2258] T 95096e4292e644e5ba5c6bee1d6ff10c P 1fe87973db4149b387542b989c096f35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 23:01:28.171190 23849 ts_tablet_manager.cc:1905] T 95096e4292e644e5ba5c6bee1d6ff10c P 1fe87973db4149b387542b989c096f35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 23:01:28.171198 23850 ts_tablet_manager.cc:1905] T 95096e4292e644e5ba5c6bee1d6ff10c P de834fd21a92405981281fa51418d325: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 23:01:28.179697 23850 ts_tablet_manager.cc:1918] T 95096e4292e644e5ba5c6bee1d6ff10c P de834fd21a92405981281fa51418d325: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 23:01:28.179955 23850 log.cc:1196] T 95096e4292e644e5ba5c6bee1d6ff10c P de834fd21a92405981281fa51418d325: Deleting WAL directory at /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786143361779964-25976-0/minicluster-data/ts-2/wal/wals/95096e4292e644e5ba5c6bee1d6ff10c
I20260807 23:01:28.180590 23850 ts_tablet_manager.cc:1939] T 95096e4292e644e5ba5c6bee1d6ff10c P de834fd21a92405981281fa51418d325: Deleting consensus metadata
I20260807 23:01:28.183496 23851 ts_tablet_manager.cc:1905] T 95096e4292e644e5ba5c6bee1d6ff10c P 70561e00ea154a38ad150af53ada25cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 23:01:28.187338 23018 catalog_manager.cc:4841] TS de834fd21a92405981281fa51418d325 (127.25.94.3:45667): tablet 95096e4292e644e5ba5c6bee1d6ff10c (table test-table [id=ecdf6fdb29334b209a5f7086519e8ca5]) successfully deleted
I20260807 23:01:28.188014 23849 ts_tablet_manager.cc:1918] T 95096e4292e644e5ba5c6bee1d6ff10c P 1fe87973db4149b387542b989c096f35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 23:01:28.188284 23849 log.cc:1196] T 95096e4292e644e5ba5c6bee1d6ff10c P 1fe87973db4149b387542b989c096f35: Deleting WAL directory at /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786143361779964-25976-0/minicluster-data/ts-0/wal/wals/95096e4292e644e5ba5c6bee1d6ff10c
I20260807 23:01:28.188942 23849 ts_tablet_manager.cc:1939] T 95096e4292e644e5ba5c6bee1d6ff10c P 1fe87973db4149b387542b989c096f35: Deleting consensus metadata
I20260807 23:01:28.191159 23018 catalog_manager.cc:4841] TS 1fe87973db4149b387542b989c096f35 (127.25.94.1:46341): tablet 95096e4292e644e5ba5c6bee1d6ff10c (table test-table [id=ecdf6fdb29334b209a5f7086519e8ca5]) successfully deleted
I20260807 23:01:28.191468 23802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:28.154938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36078 (local address 127.25.94.1:46341)
0807 23:01:28.155199 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0807 23:01:28.155209 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 23:01:28.176350 (+ 21141us) server_negotiation.cc:407] Connection header received
0807 23:01:28.176675 (+ 325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:01:28.176681 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:01:28.176796 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:01:28.176875 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:01:28.177659 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:28.178879 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:28.181958 (+ 3079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:28.182677 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:28.182800 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:28.189467 (+ 6667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:01:28.189482 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:01:28.189502 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 23:01:28.190361 (+ 859us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:01:28.190435 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 23:01:28.190465 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:01:28.190582 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0807 23:01:28.190909 (+ 327us) server_negotiation.cc:299] Negotiation successful
0807 23:01:28.191057 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260807 23:01:28.192497 23851 ts_tablet_manager.cc:1918] T 95096e4292e644e5ba5c6bee1d6ff10c P 70561e00ea154a38ad150af53ada25cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 23:01:28.192656 23851 log.cc:1196] T 95096e4292e644e5ba5c6bee1d6ff10c P 70561e00ea154a38ad150af53ada25cc: Deleting WAL directory at /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786143361779964-25976-0/minicluster-data/ts-1/wal/wals/95096e4292e644e5ba5c6bee1d6ff10c
I20260807 23:01:28.193135 23345 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 23:01:28.193326 23851 ts_tablet_manager.cc:1939] T 95096e4292e644e5ba5c6bee1d6ff10c P 70561e00ea154a38ad150af53ada25cc: Deleting consensus metadata
I20260807 23:01:28.196678 23015 catalog_manager.cc:4841] TS 70561e00ea154a38ad150af53ada25cc (127.25.94.2:43823): tablet 95096e4292e644e5ba5c6bee1d6ff10c (table test-table [id=ecdf6fdb29334b209a5f7086519e8ca5]) successfully deleted
I20260807 23:01:28.350394 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 23138
I20260807 23:01:28.386408 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 23362
I20260807 23:01:28.397861 23800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:28.387323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54788 (local address 127.25.94.2:43823)
0807 23:01:28.387516 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0807 23:01:28.387521 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 23:01:28.387539 (+ 18us) server_negotiation.cc:407] Connection header received
0807 23:01:28.387734 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:01:28.387739 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:01:28.387833 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:01:28.387917 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:01:28.388419 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:28.389147 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:28.391092 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:28.391601 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:28.391712 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:28.396317 (+ 4605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:01:28.396327 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:01:28.396342 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 23:01:28.396964 (+ 622us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:01:28.397015 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 23:01:28.397043 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:01:28.397139 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0807 23:01:28.397355 (+ 216us) server_negotiation.cc:299] Negotiation successful
0807 23:01:28.397457 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260807 23:01:28.398627 23571 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 23:01:28.488966 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 23362
I20260807 23:01:28.517382 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 23584
I20260807 23:01:28.525724 23798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:28.518202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53088 (local address 127.25.94.3:45667)
0807 23:01:28.518406 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0807 23:01:28.518412 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 23:01:28.518436 (+ 24us) server_negotiation.cc:407] Connection header received
0807 23:01:28.518510 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:01:28.518521 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:01:28.518621 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:01:28.518748 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:01:28.519210 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:28.520406 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:28.521950 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:28.522631 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:28.522783 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:28.523882 (+ 1099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:01:28.523894 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:01:28.523914 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 23:01:28.524774 (+ 860us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:01:28.524836 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 23:01:28.524862 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:01:28.525003 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0807 23:01:28.525209 (+ 206us) server_negotiation.cc:299] Negotiation successful
0807 23:01:28.525324 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260807 23:01:28.526623 23774 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 23:01:28.650920 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 23584
I20260807 23:01:28.687109 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 22992
I20260807 23:01:28.688486 23091 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 23:01:28.805550 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 22992
2026-08-07T23:01:28Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (3272 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786143361779964-25976-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 23:01:28 dist-test-slave-z8x9 krb5kdc[23870](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 23:01:28 dist-test-slave-z8x9 krb5kdc[23870](info): set up 2 sockets
Aug 07 23:01:28 dist-test-slave-z8x9 krb5kdc[23870](info): commencing operation
krb5kdc: starting...
W20260807 23:01:30.177551 25976 mini_kdc.cc:121] Time spent starting KDC: real 1.318s user 0.006s sys 0.036s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786143361779964-25976-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786143361779964-25976-0/krb5kdc/test-user.keytab.
Aug 07 23:01:30 dist-test-slave-z8x9 krb5kdc[23870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143690, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T23:01:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T23:01:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.94.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.62.keytab.
Entry for principal kudu/127.25.94.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.62.keytab.
I20260807 23:01:30.499213 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786143361779964-25976-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786143361779964-25976-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786143361779964-25976-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.62:44451
--webserver_interface=127.25.94.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.62.keytab
--principal=kudu/127.25.94.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.20:36129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.94.62:44451
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786143361779964-25976-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786143361779964-25976-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786143361779964-25976-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 23:01:30.848099 23889 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 23:01:30.848481 23889 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 23:01:30.848651 23889 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 23:01:30.856264 23889 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 23:01:30.856432 23889 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 23:01:30.856546 23889 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 23:01:30.856660 23889 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 23:01:30.856753 23889 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 23:01:30.867228 23889 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:36129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786143361779964-25976-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786143361779964-25976-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.94.62:44451
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.62.keytab
--principal=kudu/127.25.94.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.62:44451
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.94.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786143361779964-25976-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 23:01:30.868680 23889 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 23:01:30.870100 23889 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 23:01:30.882155 23895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 23:01:30.882243 23896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 23:01:30.882201 23898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 23:01:30.883319 23889 server_base.cc:1034] running on GCE node
I20260807 23:01:30.884140 23889 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 23:01:30.885674 23889 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 23:01:30.886926 23889 hybrid_clock.cc:648] HybridClock initialized: now 1786143690886879 us; error 64 us; skew 500 ppm
I20260807 23:01:30.887290 23889 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 23:01:30 dist-test-slave-z8x9 krb5kdc[23870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143690, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 23:01:30.890627 23889 init.cc:377] Logged in from keytab as kudu/127.25.94.62@KRBTEST.COM (short username kudu)
I20260807 23:01:30.892158 23889 webserver.cc:460] Webserver started at http://127.25.94.62:41965/ using document root <none> and password file <none>
I20260807 23:01:30.892884 23889 fs_manager.cc:359] Metadata directory not provided
I20260807 23:01:30.893040 23889 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 23:01:30.893422 23889 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 23:01:30.896459 23889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786143361779964-25976-0/minicluster-data/master-0/data/instance:
uuid: "eaa76a92ba624f99a472d502538706b4"
format_stamp: "Formatted at 2026-08-07 23:01:30 on dist-test-slave-z8x9"
server_key: "71763686ce81aaf9c1282e6fd6552bc2"
server_key_iv: "d28e1030de2fbe5e02814738646da7bf"
server_key_version: "encryptionkey@0"
I20260807 23:01:30.897346 23889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786143361779964-25976-0/minicluster-data/master-0/wal/instance:
uuid: "eaa76a92ba624f99a472d502538706b4"
format_stamp: "Formatted at 2026-08-07 23:01:30 on dist-test-slave-z8x9"
server_key: "71763686ce81aaf9c1282e6fd6552bc2"
server_key_iv: "d28e1030de2fbe5e02814738646da7bf"
server_key_version: "encryptionkey@0"
I20260807 23:01:30.902704 23889 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260807 23:01:30.906296 23910 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 23:01:30.907646 23889 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 23:01:30.907851 23889 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786143361779964-25976-0/minicluster-data/master-0/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786143361779964-25976-0/minicluster-data/master-0/wal
uuid: "eaa76a92ba624f99a472d502538706b4"
format_stamp: "Formatted at 2026-08-07 23:01:30 on dist-test-slave-z8x9"
server_key: "71763686ce81aaf9c1282e6fd6552bc2"
server_key_iv: "d28e1030de2fbe5e02814738646da7bf"
server_key_version: "encryptionkey@0"
I20260807 23:01:30.908057 23889 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786143361779964-25976-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786143361779964-25976-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786143361779964-25976-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 23:01:30.924197 23889 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 23:01:30.929299 23889 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 23:01:30.929636 23889 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 23:01:30.949780 23889 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.62:44451
I20260807 23:01:30.949880 23990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.62:44451 every 8 connection(s)
I20260807 23:01:30.951256 23889 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
I20260807 23:01:30.955168 23992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 23:01:30.959743 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 23889
I20260807 23:01:30.959956 25976 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786143361779964-25976-0/minicluster-data/master-0/wal/instance
I20260807 23:01:30.960453 25976 external_mini_cluster.cc:1428] Setting key 5b5c1cace4ab80d3eb020445fc7f01e8
I20260807 23:01:30.967949 23992 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eaa76a92ba624f99a472d502538706b4: Bootstrap starting.
I20260807 23:01:30.971709 23992 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eaa76a92ba624f99a472d502538706b4: Neither blocks nor log segments found. Creating new log.
I20260807 23:01:30.972903 23992 log.cc:824] T 00000000000000000000000000000000 P eaa76a92ba624f99a472d502538706b4: Log is configured to *not* fsync() on all Append() calls
I20260807 23:01:30.975823 23992 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eaa76a92ba624f99a472d502538706b4: No bootstrap required, opened a new log
Aug 07 23:01:30 dist-test-slave-z8x9 krb5kdc[23870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143690, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.62@KRBTEST.COM
I20260807 23:01:30.986217 23992 raft_consensus.cc:348] T 00000000000000000000000000000000 P eaa76a92ba624f99a472d502538706b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eaa76a92ba624f99a472d502538706b4" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 44451 } }
I20260807 23:01:30.986696 23992 raft_consensus.cc:374] T 00000000000000000000000000000000 P eaa76a92ba624f99a472d502538706b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 23:01:30.986829 23992 raft_consensus.cc:729] T 00000000000000000000000000000000 P eaa76a92ba624f99a472d502538706b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaa76a92ba624f99a472d502538706b4, State: Initialized, Role: FOLLOWER
I20260807 23:01:30.987531 23992 consensus_queue.cc:260] T 00000000000000000000000000000000 P eaa76a92ba624f99a472d502538706b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eaa76a92ba624f99a472d502538706b4" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 44451 } }
I20260807 23:01:30.987775 23996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:30.962734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45234 (local address 127.25.94.62:44451)
0807 23:01:30.963672 (+ 938us) server_negotiation.cc:206] Beginning negotiation
0807 23:01:30.963689 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0807 23:01:30.963745 (+ 56us) server_negotiation.cc:407] Connection header received
0807 23:01:30.966369 (+ 2624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:01:30.966400 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:01:30.966894 (+ 494us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:01:30.967361 (+ 467us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:01:30.968812 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:30.970908 (+ 2096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:30.972456 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:30.973624 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:30.973778 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:30.978432 (+ 4654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:01:30.978444 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:01:30.978470 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 23:01:30.978514 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:01:30.982719 (+ 4205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:01:30.983525 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:01:30.983539 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:01:30.983549 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:01:30.983779 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:01:30.984175 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:01:30.984180 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:01:30.984183 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:01:30.985000 (+ 817us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:01:30.985322 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0807 23:01:30.985785 (+ 463us) server_negotiation.cc:299] Negotiation successful
0807 23:01:30.986254 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":665,"thread_start_us":389,"threads_started":1}
I20260807 23:01:30.987854 23992 raft_consensus.cc:388] T 00000000000000000000000000000000 P eaa76a92ba624f99a472d502538706b4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 23:01:30.987983 23992 raft_consensus.cc:482] T 00000000000000000000000000000000 P eaa76a92ba624f99a472d502538706b4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 23:01:30.988186 23992 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eaa76a92ba624f99a472d502538706b4 [term 0 FOLLOWER]: Advancing to term 1
I20260807 23:01:30.990777 23992 raft_consensus.cc:504] T 00000000000000000000000000000000 P eaa76a92ba624f99a472d502538706b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eaa76a92ba624f99a472d502538706b4" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 44451 } }
I20260807 23:01:30.991361 23992 leader_election.cc:302] T 00000000000000000000000000000000 P eaa76a92ba624f99a472d502538706b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eaa76a92ba624f99a472d502538706b4; no voters:
I20260807 23:01:30.991927 23992 leader_election.cc:288] T 00000000000000000000000000000000 P eaa76a92ba624f99a472d502538706b4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 23:01:30.992098 24000 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eaa76a92ba624f99a472d502538706b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 23:01:30.992832 24000 raft_consensus.cc:686] T 00000000000000000000000000000000 P eaa76a92ba624f99a472d502538706b4 [term 1 LEADER]: Becoming Leader. State: Replica: eaa76a92ba624f99a472d502538706b4, State: Running, Role: LEADER
I20260807 23:01:30.993503 24000 consensus_queue.cc:237] T 00000000000000000000000000000000 P eaa76a92ba624f99a472d502538706b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eaa76a92ba624f99a472d502538706b4" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 44451 } }
I20260807 23:01:30.994174 23992 sys_catalog.cc:564] T 00000000000000000000000000000000 P eaa76a92ba624f99a472d502538706b4 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 23:01:30.996569 24002 sys_catalog.cc:455] T 00000000000000000000000000000000 P eaa76a92ba624f99a472d502538706b4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eaa76a92ba624f99a472d502538706b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eaa76a92ba624f99a472d502538706b4" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 44451 } } }
I20260807 23:01:30.996500 24004 sys_catalog.cc:455] T 00000000000000000000000000000000 P eaa76a92ba624f99a472d502538706b4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eaa76a92ba624f99a472d502538706b4. Latest consensus state: current_term: 1 leader_uuid: "eaa76a92ba624f99a472d502538706b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eaa76a92ba624f99a472d502538706b4" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 44451 } } }
I20260807 23:01:30.996888 24002 sys_catalog.cc:458] T 00000000000000000000000000000000 P eaa76a92ba624f99a472d502538706b4 [sys.catalog]: This master's current role is: LEADER
I20260807 23:01:30.996888 24004 sys_catalog.cc:458] T 00000000000000000000000000000000 P eaa76a92ba624f99a472d502538706b4 [sys.catalog]: This master's current role is: LEADER
I20260807 23:01:31.001340 24014 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 23:01:31.008033 24014 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 23:01:31.017256 24014 catalog_manager.cc:1347] Generated new cluster ID: 6d1b3c154b2746a9918092282a675ea2
I20260807 23:01:31.017395 24014 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 23:01:31.032514 24014 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 23:01:31.034480 24014 catalog_manager.cc:1504] Loading token signing keys...
I20260807 23:01:31.051339 24014 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eaa76a92ba624f99a472d502538706b4: Generated new TSK 0
I20260807 23:01:31.052217 24014 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.94.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.1.keytab.
Entry for principal kudu/127.25.94.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.1.keytab.
I20260807 23:01:31.133704 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786143361779964-25976-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786143361779964-25976-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786143361779964-25976-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786143361779964-25976-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.1:0
--local_ip_for_outbound_sockets=127.25.94.1
--webserver_interface=127.25.94.1
--webserver_port=0
--tserver_master_addrs=127.25.94.62:44451
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.1.keytab
--principal=kudu/127.25.94.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.20:36129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786143361779964-25976-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786143361779964-25976-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786143361779964-25976-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 23:01:31.498193 24042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 23:01:31.498538 24042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 23:01:31.498708 24042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 23:01:31.506426 24042 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 23:01:31.506589 24042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 23:01:31.506760 24042 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.1
I20260807 23:01:31.517060 24042 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:36129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786143361779964-25976-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786143361779964-25976-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.1.keytab
--principal=kudu/127.25.94.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786143361779964-25976-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.94.1
--webserver_port=0
--tserver_master_addrs=127.25.94.62:44451
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.1
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786143361779964-25976-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 23:01:31.518467 24042 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 23:01:31.519775 24042 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 23:01:31.533298 24048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 23:01:31.533404 24047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 23:01:31.533380 24051 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 23:01:31.534729 24042 server_base.cc:1034] running on GCE node
I20260807 23:01:31.535377 24042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 23:01:31.536334 24042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 23:01:31.537586 24042 hybrid_clock.cc:648] HybridClock initialized: now 1786143691537542 us; error 52 us; skew 500 ppm
I20260807 23:01:31.537927 24042 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 23:01:31 dist-test-slave-z8x9 krb5kdc[23870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143691, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 23:01:31.541101 24042 init.cc:377] Logged in from keytab as kudu/127.25.94.1@KRBTEST.COM (short username kudu)
I20260807 23:01:31.542691 24042 webserver.cc:460] Webserver started at http://127.25.94.1:41955/ using document root <none> and password file <none>
I20260807 23:01:31.543424 24042 fs_manager.cc:359] Metadata directory not provided
I20260807 23:01:31.543574 24042 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 23:01:31.543920 24042 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 23:01:31.546918 24042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786143361779964-25976-0/minicluster-data/ts-0/data/instance:
uuid: "b8b4c95e9c12440eada45ae22e226f9f"
format_stamp: "Formatted at 2026-08-07 23:01:31 on dist-test-slave-z8x9"
server_key: "7abfd48c3d219925a57ab4522b44bdf4"
server_key_iv: "998ec5a0f7a2bc115623775bb638de4e"
server_key_version: "encryptionkey@0"
I20260807 23:01:31.547710 24042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786143361779964-25976-0/minicluster-data/ts-0/wal/instance:
uuid: "b8b4c95e9c12440eada45ae22e226f9f"
format_stamp: "Formatted at 2026-08-07 23:01:31 on dist-test-slave-z8x9"
server_key: "7abfd48c3d219925a57ab4522b44bdf4"
server_key_iv: "998ec5a0f7a2bc115623775bb638de4e"
server_key_version: "encryptionkey@0"
I20260807 23:01:31.553123 24042 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260807 23:01:31.556705 24067 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 23:01:31.558041 24042 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 23:01:31.558229 24042 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786143361779964-25976-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786143361779964-25976-0/minicluster-data/ts-0/wal
uuid: "b8b4c95e9c12440eada45ae22e226f9f"
format_stamp: "Formatted at 2026-08-07 23:01:31 on dist-test-slave-z8x9"
server_key: "7abfd48c3d219925a57ab4522b44bdf4"
server_key_iv: "998ec5a0f7a2bc115623775bb638de4e"
server_key_version: "encryptionkey@0"
I20260807 23:01:31.558429 24042 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786143361779964-25976-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786143361779964-25976-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786143361779964-25976-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 23:01:31.582865 24042 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 23:01:31.587235 24042 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 23:01:31.587651 24042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 23:01:31.588738 24042 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 23:01:31.590126 24042 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 23:01:31.590237 24042 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 23:01:31.590387 24042 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 23:01:31.590463 24042 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 23:01:31.625965 24042 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.1:42875
I20260807 23:01:31.625998 24246 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.1:42875 every 8 connection(s)
I20260807 23:01:31.627341 24042 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786143361779964-25976-0/minicluster-data/ts-0/data/info.pb
I20260807 23:01:31.635841 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 24042
I20260807 23:01:31.636065 25976 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786143361779964-25976-0/minicluster-data/ts-0/wal/instance
I20260807 23:01:31.636539 25976 external_mini_cluster.cc:1428] Setting key 5095fea6170bb30f8f509e78016e97de
Aug 07 23:01:31 dist-test-slave-z8x9 krb5kdc[23870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143691, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.1@KRBTEST.COM for kudu/127.25.94.62@KRBTEST.COM
I20260807 23:01:31.648794 24249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:31.630068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.1:58013 (local address 127.25.94.62:44451)
0807 23:01:31.630555 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0807 23:01:31.630560 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 23:01:31.631964 (+ 1404us) server_negotiation.cc:407] Connection header received
0807 23:01:31.634020 (+ 2056us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:01:31.634026 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:01:31.634126 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:01:31.634250 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:01:31.635818 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:31.636725 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:31.638380 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:31.638795 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:31.638951 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:31.642526 (+ 3575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:01:31.642531 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:01:31.642534 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 23:01:31.642564 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:01:31.645559 (+ 2995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:01:31.646372 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:01:31.646377 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:01:31.646379 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:01:31.646500 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:01:31.646958 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:01:31.646962 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:01:31.646972 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:01:31.647244 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:01:31.647414 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0807 23:01:31.648189 (+ 775us) server_negotiation.cc:299] Negotiation successful
0807 23:01:31.648431 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":233,"threads_started":1}
I20260807 23:01:31.649852 24251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:31.630646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:44451 (local address 127.25.94.1:58013)
0807 23:01:31.631761 (+ 1115us) negotiation.cc:107] Waiting for socket to connect
0807 23:01:31.631807 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0807 23:01:31.633713 (+ 1906us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 23:01:31.634504 (+ 791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 23:01:31.634512 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 23:01:31.635061 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0807 23:01:31.635640 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:01:31.635657 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:01:31.636885 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:01:31.636897 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:01:31.638225 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:01:31.638238 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:01:31.639048 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:01:31.639057 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:01:31.639288 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:31.640017 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 23:01:31.640039 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0807 23:01:31.642297 (+ 2258us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 23:01:31.645721 (+ 3424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 23:01:31.645731 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 23:01:31.645749 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0807 23:01:31.646215 (+ 466us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 23:01:31.646615 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 23:01:31.646620 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 23:01:31.646623 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 23:01:31.646831 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 23:01:31.647395 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 23:01:31.647404 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 23:01:31.647761 (+ 357us) client_negotiation.cc:769] Sending connection context
0807 23:01:31.648044 (+ 283us) client_negotiation.cc:240] Negotiation successful
0807 23:01:31.648597 (+ 553us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":712,"thread_start_us":480,"threads_started":1}
I20260807 23:01:31.651425 24247 heartbeater.cc:347] Connected to a master server at 127.25.94.62:44451
I20260807 23:01:31.651769 24247 heartbeater.cc:464] Registering TS with master...
I20260807 23:01:31.652649 24247 heartbeater.cc:511] Master 127.25.94.62:44451 requested a full tablet report, sending...
I20260807 23:01:31.655136 23936 ts_manager.cc:194] Registered new tserver with Master: b8b4c95e9c12440eada45ae22e226f9f (127.25.94.1:42875)
I20260807 23:01:31.656651 23936 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.94.1@KRBTEST.COM'} at 127.25.94.1:58013
WARNING: no policy specified for kudu/127.25.94.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.2.keytab.
Entry for principal kudu/127.25.94.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.2.keytab.
I20260807 23:01:31.704735 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786143361779964-25976-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786143361779964-25976-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786143361779964-25976-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786143361779964-25976-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.2:0
--local_ip_for_outbound_sockets=127.25.94.2
--webserver_interface=127.25.94.2
--webserver_port=0
--tserver_master_addrs=127.25.94.62:44451
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.2.keytab
--principal=kudu/127.25.94.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.20:36129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786143361779964-25976-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786143361779964-25976-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786143361779964-25976-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 23:01:32.051141 24255 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 23:01:32.051498 24255 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 23:01:32.051687 24255 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 23:01:32.059087 24255 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 23:01:32.059266 24255 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 23:01:32.059445 24255 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.2
I20260807 23:01:32.069504 24255 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:36129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786143361779964-25976-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786143361779964-25976-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.2.keytab
--principal=kudu/127.25.94.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786143361779964-25976-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.94.2
--webserver_port=0
--tserver_master_addrs=127.25.94.62:44451
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.2
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786143361779964-25976-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 23:01:32.070900 24255 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 23:01:32.072250 24255 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 23:01:32.085891 24264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 23:01:32.085901 24266 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 23:01:32.085940 24268 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 23:01:32.087060 24255 server_base.cc:1034] running on GCE node
I20260807 23:01:32.087745 24255 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 23:01:32.088775 24255 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 23:01:32.090112 24255 hybrid_clock.cc:648] HybridClock initialized: now 1786143692090059 us; error 50 us; skew 500 ppm
I20260807 23:01:32.090467 24255 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 23:01:32 dist-test-slave-z8x9 krb5kdc[23870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143692, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 23:01:32.093940 24255 init.cc:377] Logged in from keytab as kudu/127.25.94.2@KRBTEST.COM (short username kudu)
I20260807 23:01:32.095587 24255 webserver.cc:460] Webserver started at http://127.25.94.2:45801/ using document root <none> and password file <none>
I20260807 23:01:32.096251 24255 fs_manager.cc:359] Metadata directory not provided
I20260807 23:01:32.096364 24255 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 23:01:32.096635 24255 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 23:01:32.099694 24255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786143361779964-25976-0/minicluster-data/ts-1/data/instance:
uuid: "484ca854ed3e44f4a26e1fcbee681427"
format_stamp: "Formatted at 2026-08-07 23:01:32 on dist-test-slave-z8x9"
server_key: "116faf3f0f6540e6eb003bd9eacad102"
server_key_iv: "58633542fae3a8feee1c8f3144e62e5e"
server_key_version: "encryptionkey@0"
I20260807 23:01:32.100461 24255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786143361779964-25976-0/minicluster-data/ts-1/wal/instance:
uuid: "484ca854ed3e44f4a26e1fcbee681427"
format_stamp: "Formatted at 2026-08-07 23:01:32 on dist-test-slave-z8x9"
server_key: "116faf3f0f6540e6eb003bd9eacad102"
server_key_iv: "58633542fae3a8feee1c8f3144e62e5e"
server_key_version: "encryptionkey@0"
I20260807 23:01:32.105957 24255 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260807 23:01:32.109760 24280 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 23:01:32.111285 24255 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260807 23:01:32.111515 24255 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786143361779964-25976-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786143361779964-25976-0/minicluster-data/ts-1/wal
uuid: "484ca854ed3e44f4a26e1fcbee681427"
format_stamp: "Formatted at 2026-08-07 23:01:32 on dist-test-slave-z8x9"
server_key: "116faf3f0f6540e6eb003bd9eacad102"
server_key_iv: "58633542fae3a8feee1c8f3144e62e5e"
server_key_version: "encryptionkey@0"
I20260807 23:01:32.111788 24255 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786143361779964-25976-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786143361779964-25976-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786143361779964-25976-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 23:01:32.135053 24255 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 23:01:32.140046 24255 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 23:01:32.140565 24255 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 23:01:32.142000 24255 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 23:01:32.143540 24255 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 23:01:32.143630 24255 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 23:01:32.143787 24255 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 23:01:32.143877 24255 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 23:01:32.181470 24255 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.2:43709
I20260807 23:01:32.181572 24457 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.2:43709 every 8 connection(s)
I20260807 23:01:32.182869 24255 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786143361779964-25976-0/minicluster-data/ts-1/data/info.pb
I20260807 23:01:32.186493 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 24255
I20260807 23:01:32.186723 25976 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786143361779964-25976-0/minicluster-data/ts-1/wal/instance
I20260807 23:01:32.187364 25976 external_mini_cluster.cc:1428] Setting key 3b458515254f6accc12a11f3c0e0fb28
Aug 07 23:01:32 dist-test-slave-z8x9 krb5kdc[23870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143692, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.2@KRBTEST.COM for kudu/127.25.94.62@KRBTEST.COM
I20260807 23:01:32.205554 24461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:32.186091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.2:45265 (local address 127.25.94.62:44451)
0807 23:01:32.186681 (+ 590us) server_negotiation.cc:206] Beginning negotiation
0807 23:01:32.186689 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 23:01:32.187910 (+ 1221us) server_negotiation.cc:407] Connection header received
0807 23:01:32.190043 (+ 2133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:01:32.190049 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:01:32.190144 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:01:32.190267 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:01:32.191901 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:32.192660 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:32.194428 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:32.194882 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:32.195073 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:32.199200 (+ 4127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:01:32.199205 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:01:32.199208 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 23:01:32.199250 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:01:32.202193 (+ 2943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:01:32.203028 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:01:32.203040 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:01:32.203043 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:01:32.203182 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:01:32.203685 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:01:32.203694 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:01:32.203698 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:01:32.203982 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:01:32.204142 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0807 23:01:32.204883 (+ 741us) server_negotiation.cc:299] Negotiation successful
0807 23:01:32.205130 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":326,"threads_started":1}
I20260807 23:01:32.206609 24462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:32.186842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:44451 (local address 127.25.94.2:45265)
0807 23:01:32.187741 (+ 899us) negotiation.cc:107] Waiting for socket to connect
0807 23:01:32.187779 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0807 23:01:32.189762 (+ 1983us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 23:01:32.190529 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 23:01:32.190537 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 23:01:32.191102 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0807 23:01:32.191702 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:01:32.191721 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:01:32.192844 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:01:32.192848 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:01:32.194278 (+ 1430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:01:32.194293 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:01:32.195126 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:01:32.195135 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:01:32.195414 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:32.196171 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 23:01:32.196195 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0807 23:01:32.198914 (+ 2719us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 23:01:32.202362 (+ 3448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 23:01:32.202372 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 23:01:32.202391 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0807 23:01:32.202860 (+ 469us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 23:01:32.203320 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 23:01:32.203326 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 23:01:32.203329 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 23:01:32.203554 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 23:01:32.204134 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 23:01:32.204142 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 23:01:32.204496 (+ 354us) client_negotiation.cc:769] Sending connection context
0807 23:01:32.204782 (+ 286us) client_negotiation.cc:240] Negotiation successful
0807 23:01:32.205404 (+ 622us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":653,"thread_start_us":353,"threads_started":1}
I20260807 23:01:32.208210 24458 heartbeater.cc:347] Connected to a master server at 127.25.94.62:44451
I20260807 23:01:32.208563 24458 heartbeater.cc:464] Registering TS with master...
I20260807 23:01:32.209370 24458 heartbeater.cc:511] Master 127.25.94.62:44451 requested a full tablet report, sending...
I20260807 23:01:32.211022 23936 ts_manager.cc:194] Registered new tserver with Master: 484ca854ed3e44f4a26e1fcbee681427 (127.25.94.2:43709)
I20260807 23:01:32.212323 23936 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.94.2@KRBTEST.COM'} at 127.25.94.2:45265
WARNING: no policy specified for kudu/127.25.94.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.3.keytab.
Entry for principal kudu/127.25.94.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.3.keytab.
I20260807 23:01:32.261372 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786143361779964-25976-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786143361779964-25976-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786143361779964-25976-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786143361779964-25976-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.3:0
--local_ip_for_outbound_sockets=127.25.94.3
--webserver_interface=127.25.94.3
--webserver_port=0
--tserver_master_addrs=127.25.94.62:44451
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.3.keytab
--principal=kudu/127.25.94.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.20:36129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786143361779964-25976-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786143361779964-25976-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786143361779964-25976-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 23:01:32.612901 24465 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 23:01:32.613269 24465 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 23:01:32.613441 24465 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 23:01:32.620873 24465 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 23:01:32.621047 24465 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 23:01:32.621249 24465 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.3
I20260807 23:01:32.631269 24465 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:36129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786143361779964-25976-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786143361779964-25976-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.3.keytab
--principal=kudu/127.25.94.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786143361779964-25976-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.94.3
--webserver_port=0
--tserver_master_addrs=127.25.94.62:44451
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.3
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786143361779964-25976-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 23:01:32.632514 24465 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 23:01:32.633819 24465 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 23:01:32.647213 24471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 23:01:32.647135 24472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 23:01:32.647169 24475 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 23:01:32.648115 24465 server_base.cc:1034] running on GCE node
I20260807 23:01:32.648840 24465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 23:01:32.649914 24465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 23:01:32.651134 24465 hybrid_clock.cc:648] HybridClock initialized: now 1786143692651072 us; error 56 us; skew 500 ppm
I20260807 23:01:32.651453 24465 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 23:01:32 dist-test-slave-z8x9 krb5kdc[23870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143692, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 23:01:32.654953 24465 init.cc:377] Logged in from keytab as kudu/127.25.94.3@KRBTEST.COM (short username kudu)
I20260807 23:01:32.656474 24465 webserver.cc:460] Webserver started at http://127.25.94.3:35723/ using document root <none> and password file <none>
I20260807 23:01:32.657403 24465 fs_manager.cc:359] Metadata directory not provided
I20260807 23:01:32.657544 24465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 23:01:32.657889 24465 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 23:01:32.660739 24247 heartbeater.cc:503] Master 127.25.94.62:44451 was elected leader, sending a full tablet report...
I20260807 23:01:32.661607 24465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786143361779964-25976-0/minicluster-data/ts-2/data/instance:
uuid: "dcbc18964e3d49118bb745a1f89bd76f"
format_stamp: "Formatted at 2026-08-07 23:01:32 on dist-test-slave-z8x9"
server_key: "6717ca4b859523f7dda9d52ea7db01f2"
server_key_iv: "34d6b328673d07db2b5db5b648feb8d5"
server_key_version: "encryptionkey@0"
I20260807 23:01:32.662774 24465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786143361779964-25976-0/minicluster-data/ts-2/wal/instance:
uuid: "dcbc18964e3d49118bb745a1f89bd76f"
format_stamp: "Formatted at 2026-08-07 23:01:32 on dist-test-slave-z8x9"
server_key: "6717ca4b859523f7dda9d52ea7db01f2"
server_key_iv: "34d6b328673d07db2b5db5b648feb8d5"
server_key_version: "encryptionkey@0"
I20260807 23:01:32.674958 24465 fs_manager.cc:691] Time spent creating directory manager: real 0.012s user 0.008s sys 0.004s
I20260807 23:01:32.679186 24487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 23:01:32.680884 24465 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260807 23:01:32.681041 24465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786143361779964-25976-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786143361779964-25976-0/minicluster-data/ts-2/wal
uuid: "dcbc18964e3d49118bb745a1f89bd76f"
format_stamp: "Formatted at 2026-08-07 23:01:32 on dist-test-slave-z8x9"
server_key: "6717ca4b859523f7dda9d52ea7db01f2"
server_key_iv: "34d6b328673d07db2b5db5b648feb8d5"
server_key_version: "encryptionkey@0"
I20260807 23:01:32.681309 24465 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786143361779964-25976-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786143361779964-25976-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786143361779964-25976-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 23:01:32.719691 24465 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 23:01:32.724463 24465 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 23:01:32.724735 24465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 23:01:32.725824 24465 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 23:01:32.727279 24465 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 23:01:32.727368 24465 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 23:01:32.727546 24465 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 23:01:32.727602 24465 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 23:01:32.763654 24465 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.3:43771
I20260807 23:01:32.763695 24654 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.3:43771 every 8 connection(s)
I20260807 23:01:32.765017 24465 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786143361779964-25976-0/minicluster-data/ts-2/data/info.pb
I20260807 23:01:32.767043 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 24465
I20260807 23:01:32.767259 25976 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786143361779964-25976-0/minicluster-data/ts-2/wal/instance
I20260807 23:01:32.767872 25976 external_mini_cluster.cc:1428] Setting key 4d3de061afbf09ddf783ff048df12bd8
Aug 07 23:01:32 dist-test-slave-z8x9 krb5kdc[23870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143692, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.3@KRBTEST.COM for kudu/127.25.94.62@KRBTEST.COM
I20260807 23:01:32.786660 24659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:32.768125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.3:41983 (local address 127.25.94.62:44451)
0807 23:01:32.768661 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0807 23:01:32.768666 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 23:01:32.769917 (+ 1251us) server_negotiation.cc:407] Connection header received
0807 23:01:32.772059 (+ 2142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:01:32.772065 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:01:32.772157 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:01:32.772273 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:01:32.773906 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:32.774687 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:32.776239 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:32.776656 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:32.776799 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:32.780358 (+ 3559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:01:32.780362 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:01:32.780366 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 23:01:32.780398 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:01:32.783671 (+ 3273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:01:32.784365 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:01:32.784369 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:01:32.784371 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:01:32.784496 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:01:32.784957 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:01:32.784961 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:01:32.784963 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:01:32.785313 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:01:32.785433 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0807 23:01:32.786104 (+ 671us) server_negotiation.cc:299] Negotiation successful
0807 23:01:32.786310 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":289,"threads_started":1}
I20260807 23:01:32.787617 24660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:32.768740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:44451 (local address 127.25.94.3:41983)
0807 23:01:32.769735 (+ 995us) negotiation.cc:107] Waiting for socket to connect
0807 23:01:32.769790 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0807 23:01:32.771787 (+ 1997us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 23:01:32.772529 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 23:01:32.772538 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 23:01:32.773078 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0807 23:01:32.773727 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:01:32.773746 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:01:32.774858 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:01:32.774862 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:01:32.776091 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:01:32.776103 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:01:32.776899 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:01:32.776907 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:01:32.777142 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:32.777955 (+ 813us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 23:01:32.777978 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0807 23:01:32.780160 (+ 2182us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 23:01:32.783803 (+ 3643us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 23:01:32.783810 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 23:01:32.783822 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0807 23:01:32.784248 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 23:01:32.784636 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 23:01:32.784639 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 23:01:32.784641 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 23:01:32.784850 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 23:01:32.785419 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 23:01:32.785426 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 23:01:32.785745 (+ 319us) client_negotiation.cc:769] Sending connection context
0807 23:01:32.785984 (+ 239us) client_negotiation.cc:240] Negotiation successful
0807 23:01:32.786450 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":702,"thread_start_us":480,"threads_started":1}
I20260807 23:01:32.789110 24656 heartbeater.cc:347] Connected to a master server at 127.25.94.62:44451
I20260807 23:01:32.789492 24656 heartbeater.cc:464] Registering TS with master...
I20260807 23:01:32.790200 24656 heartbeater.cc:511] Master 127.25.94.62:44451 requested a full tablet report, sending...
I20260807 23:01:32.791563 23936 ts_manager.cc:194] Registered new tserver with Master: dcbc18964e3d49118bb745a1f89bd76f (127.25.94.3:43771)
I20260807 23:01:32.792335 23936 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.94.3@KRBTEST.COM'} at 127.25.94.3:41983
I20260807 23:01:32.795746 25976 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 23:01:32.814352 24659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:32.799905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52370 (local address 127.25.94.62:44451)
0807 23:01:32.800165 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0807 23:01:32.800215 (+ 50us) server_negotiation.cc:399] Waiting for connection header
0807 23:01:32.800499 (+ 284us) server_negotiation.cc:407] Connection header received
0807 23:01:32.800616 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:01:32.800620 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:01:32.800683 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:01:32.800765 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:01:32.802293 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:32.803204 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:32.804613 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:32.805000 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:32.805099 (+ 99us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:32.808099 (+ 3000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:01:32.808104 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:01:32.808108 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 23:01:32.808142 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:01:32.810971 (+ 2829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:01:32.812129 (+ 1158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:01:32.812134 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:01:32.812137 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:01:32.812311 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:01:32.812771 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:01:32.812775 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:01:32.812777 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:01:32.813256 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:01:32.813475 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0807 23:01:32.813833 (+ 358us) server_negotiation.cc:299] Negotiation successful
0807 23:01:32.813980 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260807 23:01:32.818856 23936 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52370:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 23:01:32.825932 23936 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 23:01:32.851548 24673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:32.839131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50098 (local address 127.25.94.2:43709)
0807 23:01:32.841269 (+ 2138us) server_negotiation.cc:206] Beginning negotiation
0807 23:01:32.841281 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 23:01:32.841329 (+ 48us) server_negotiation.cc:407] Connection header received
0807 23:01:32.841460 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:01:32.841471 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:01:32.841767 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:01:32.842881 (+ 1114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 23:01:32.843160 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:32.845132 (+ 1972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:32.848667 (+ 3535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:32.850177 (+ 1510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:32.850348 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:32.850550 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0807 23:01:32.851006 (+ 456us) server_negotiation.cc:299] Negotiation successful
0807 23:01:32.851195 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1901,"thread_start_us":1768,"threads_started":1}
I20260807 23:01:32.851630 24671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:32.838940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.2:43709 (local address 127.0.0.1:50098)
0807 23:01:32.839688 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0807 23:01:32.839737 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0807 23:01:32.839902 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 23:01:32.842057 (+ 2155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 23:01:32.842065 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 23:01:32.842094 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 23:01:32.842618 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:01:32.842635 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:01:32.845313 (+ 2678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:01:32.845318 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:01:32.848383 (+ 3065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:01:32.848407 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:01:32.850373 (+ 1966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:01:32.850380 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:01:32.850585 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:32.850668 (+ 83us) client_negotiation.cc:769] Sending connection context
0807 23:01:32.850823 (+ 155us) client_negotiation.cc:240] Negotiation successful
0807 23:01:32.851066 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"mutex_wait_us":1,"thread_start_us":371,"threads_started":1}
I20260807 23:01:32.855963 24352 tablet_service.cc:1467] Processing CreateTablet for tablet 09af4f448d314336ac58b26e8f862c68 (DEFAULT_TABLE table=test-table [id=1424efd7c25f4fd5985586e450ee6b47]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 23:01:32.857508 24352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09af4f448d314336ac58b26e8f862c68. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 23:01:32.858282 24678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:32.841504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38060 (local address 127.25.94.1:42875)
0807 23:01:32.849474 (+ 7970us) server_negotiation.cc:206] Beginning negotiation
0807 23:01:32.849485 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 23:01:32.850075 (+ 590us) server_negotiation.cc:407] Connection header received
0807 23:01:32.850197 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:01:32.850206 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:01:32.850481 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:01:32.850700 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 23:01:32.851913 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:32.853074 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:32.855390 (+ 2316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:32.856834 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:32.856975 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:32.857251 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0807 23:01:32.857754 (+ 503us) server_negotiation.cc:299] Negotiation successful
0807 23:01:32.857934 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7786,"thread_start_us":7606,"threads_started":1}
I20260807 23:01:32.858462 24674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:32.840198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38288 (local address 127.25.94.3:43771)
0807 23:01:32.850196 (+ 9998us) server_negotiation.cc:206] Beginning negotiation
0807 23:01:32.850205 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 23:01:32.850234 (+ 29us) server_negotiation.cc:407] Connection header received
0807 23:01:32.850313 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:01:32.850319 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:01:32.850491 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:01:32.850692 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 23:01:32.851427 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:32.853010 (+ 1583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:32.855769 (+ 2759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:32.857280 (+ 1511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:32.857444 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:32.857674 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0807 23:01:32.857900 (+ 226us) server_negotiation.cc:299] Negotiation successful
0807 23:01:32.858057 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9821,"thread_start_us":9636,"threads_started":1}
I20260807 23:01:32.859105 24677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:32.840652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.1:42875 (local address 127.0.0.1:38060)
0807 23:01:32.849767 (+ 9115us) negotiation.cc:107] Waiting for socket to connect
0807 23:01:32.849789 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 23:01:32.849929 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 23:01:32.851017 (+ 1088us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 23:01:32.851025 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 23:01:32.851054 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 23:01:32.851480 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:01:32.851492 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:01:32.853272 (+ 1780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:01:32.853277 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:01:32.855210 (+ 1933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:01:32.855224 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:01:32.857060 (+ 1836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:01:32.857067 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:01:32.857283 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:32.857332 (+ 49us) client_negotiation.cc:769] Sending connection context
0807 23:01:32.857433 (+ 101us) client_negotiation.cc:240] Negotiation successful
0807 23:01:32.858676 (+ 1243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8984,"mutex_wait_us":25,"thread_start_us":8846,"threads_started":1}
I20260807 23:01:32.859222 24672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:32.839091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.3:43771 (local address 127.0.0.1:38288)
0807 23:01:32.840499 (+ 1408us) negotiation.cc:107] Waiting for socket to connect
0807 23:01:32.840523 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0807 23:01:32.840663 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 23:01:32.850690 (+ 10027us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 23:01:32.850695 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 23:01:32.850721 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 23:01:32.851180 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:01:32.851194 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:01:32.853219 (+ 2025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:01:32.853224 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:01:32.855581 (+ 2357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:01:32.855595 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:01:32.857505 (+ 1910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:01:32.857512 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:01:32.857694 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:32.857740 (+ 46us) client_negotiation.cc:769] Sending connection context
0807 23:01:32.857858 (+ 118us) client_negotiation.cc:240] Negotiation successful
0807 23:01:32.858755 (+ 897us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1262,"mutex_wait_us":6,"thread_start_us":2584,"threads_started":1}
I20260807 23:01:32.862802 24145 tablet_service.cc:1467] Processing CreateTablet for tablet 09af4f448d314336ac58b26e8f862c68 (DEFAULT_TABLE table=test-table [id=1424efd7c25f4fd5985586e450ee6b47]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 23:01:32.863016 24557 tablet_service.cc:1467] Processing CreateTablet for tablet 09af4f448d314336ac58b26e8f862c68 (DEFAULT_TABLE table=test-table [id=1424efd7c25f4fd5985586e450ee6b47]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 23:01:32.864126 24145 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09af4f448d314336ac58b26e8f862c68. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 23:01:32.864414 24557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09af4f448d314336ac58b26e8f862c68. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 23:01:32.873531 24680 tablet_bootstrap.cc:492] T 09af4f448d314336ac58b26e8f862c68 P 484ca854ed3e44f4a26e1fcbee681427: Bootstrap starting.
I20260807 23:01:32.875602 24684 tablet_bootstrap.cc:492] T 09af4f448d314336ac58b26e8f862c68 P b8b4c95e9c12440eada45ae22e226f9f: Bootstrap starting.
I20260807 23:01:32.877394 24680 tablet_bootstrap.cc:654] T 09af4f448d314336ac58b26e8f862c68 P 484ca854ed3e44f4a26e1fcbee681427: Neither blocks nor log segments found. Creating new log.
I20260807 23:01:32.878396 24680 log.cc:824] T 09af4f448d314336ac58b26e8f862c68 P 484ca854ed3e44f4a26e1fcbee681427: Log is configured to *not* fsync() on all Append() calls
I20260807 23:01:32.879079 24684 tablet_bootstrap.cc:654] T 09af4f448d314336ac58b26e8f862c68 P b8b4c95e9c12440eada45ae22e226f9f: Neither blocks nor log segments found. Creating new log.
I20260807 23:01:32.879436 24685 tablet_bootstrap.cc:492] T 09af4f448d314336ac58b26e8f862c68 P dcbc18964e3d49118bb745a1f89bd76f: Bootstrap starting.
I20260807 23:01:32.880451 24684 log.cc:824] T 09af4f448d314336ac58b26e8f862c68 P b8b4c95e9c12440eada45ae22e226f9f: Log is configured to *not* fsync() on all Append() calls
I20260807 23:01:32.883529 24680 tablet_bootstrap.cc:492] T 09af4f448d314336ac58b26e8f862c68 P 484ca854ed3e44f4a26e1fcbee681427: No bootstrap required, opened a new log
I20260807 23:01:32.883520 24685 tablet_bootstrap.cc:654] T 09af4f448d314336ac58b26e8f862c68 P dcbc18964e3d49118bb745a1f89bd76f: Neither blocks nor log segments found. Creating new log.
I20260807 23:01:32.883839 24684 tablet_bootstrap.cc:492] T 09af4f448d314336ac58b26e8f862c68 P b8b4c95e9c12440eada45ae22e226f9f: No bootstrap required, opened a new log
I20260807 23:01:32.883849 24680 ts_tablet_manager.cc:1397] T 09af4f448d314336ac58b26e8f862c68 P 484ca854ed3e44f4a26e1fcbee681427: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.004s
I20260807 23:01:32.884105 24684 ts_tablet_manager.cc:1397] T 09af4f448d314336ac58b26e8f862c68 P b8b4c95e9c12440eada45ae22e226f9f: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260807 23:01:32.884794 24685 log.cc:824] T 09af4f448d314336ac58b26e8f862c68 P dcbc18964e3d49118bb745a1f89bd76f: Log is configured to *not* fsync() on all Append() calls
I20260807 23:01:32.888144 24685 tablet_bootstrap.cc:492] T 09af4f448d314336ac58b26e8f862c68 P dcbc18964e3d49118bb745a1f89bd76f: No bootstrap required, opened a new log
I20260807 23:01:32.888450 24685 ts_tablet_manager.cc:1397] T 09af4f448d314336ac58b26e8f862c68 P dcbc18964e3d49118bb745a1f89bd76f: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260807 23:01:32.892194 24684 raft_consensus.cc:348] T 09af4f448d314336ac58b26e8f862c68 P b8b4c95e9c12440eada45ae22e226f9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "484ca854ed3e44f4a26e1fcbee681427" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 43709 } } peers { permanent_uuid: "dcbc18964e3d49118bb745a1f89bd76f" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 43771 } } peers { permanent_uuid: "b8b4c95e9c12440eada45ae22e226f9f" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 42875 } }
I20260807 23:01:32.892669 24684 raft_consensus.cc:374] T 09af4f448d314336ac58b26e8f862c68 P b8b4c95e9c12440eada45ae22e226f9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 23:01:32.892777 24684 raft_consensus.cc:729] T 09af4f448d314336ac58b26e8f862c68 P b8b4c95e9c12440eada45ae22e226f9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8b4c95e9c12440eada45ae22e226f9f, State: Initialized, Role: FOLLOWER
I20260807 23:01:32.893543 24684 consensus_queue.cc:260] T 09af4f448d314336ac58b26e8f862c68 P b8b4c95e9c12440eada45ae22e226f9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "484ca854ed3e44f4a26e1fcbee681427" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 43709 } } peers { permanent_uuid: "dcbc18964e3d49118bb745a1f89bd76f" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 43771 } } peers { permanent_uuid: "b8b4c95e9c12440eada45ae22e226f9f" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 42875 } }
I20260807 23:01:32.894644 24680 raft_consensus.cc:348] T 09af4f448d314336ac58b26e8f862c68 P 484ca854ed3e44f4a26e1fcbee681427 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "484ca854ed3e44f4a26e1fcbee681427" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 43709 } } peers { permanent_uuid: "dcbc18964e3d49118bb745a1f89bd76f" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 43771 } } peers { permanent_uuid: "b8b4c95e9c12440eada45ae22e226f9f" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 42875 } }
I20260807 23:01:32.895192 24680 raft_consensus.cc:374] T 09af4f448d314336ac58b26e8f862c68 P 484ca854ed3e44f4a26e1fcbee681427 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 23:01:32.895296 24680 raft_consensus.cc:729] T 09af4f448d314336ac58b26e8f862c68 P 484ca854ed3e44f4a26e1fcbee681427 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 484ca854ed3e44f4a26e1fcbee681427, State: Initialized, Role: FOLLOWER
I20260807 23:01:32.895649 24684 ts_tablet_manager.cc:1428] T 09af4f448d314336ac58b26e8f862c68 P b8b4c95e9c12440eada45ae22e226f9f: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260807 23:01:32.896265 24680 consensus_queue.cc:260] T 09af4f448d314336ac58b26e8f862c68 P 484ca854ed3e44f4a26e1fcbee681427 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "484ca854ed3e44f4a26e1fcbee681427" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 43709 } } peers { permanent_uuid: "dcbc18964e3d49118bb745a1f89bd76f" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 43771 } } peers { permanent_uuid: "b8b4c95e9c12440eada45ae22e226f9f" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 42875 } }
I20260807 23:01:32.897521 24458 heartbeater.cc:503] Master 127.25.94.62:44451 was elected leader, sending a full tablet report...
I20260807 23:01:32.898205 24680 ts_tablet_manager.cc:1428] T 09af4f448d314336ac58b26e8f862c68 P 484ca854ed3e44f4a26e1fcbee681427: Time spent starting tablet: real 0.014s user 0.006s sys 0.009s
I20260807 23:01:32.899544 24685 raft_consensus.cc:348] T 09af4f448d314336ac58b26e8f862c68 P dcbc18964e3d49118bb745a1f89bd76f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "484ca854ed3e44f4a26e1fcbee681427" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 43709 } } peers { permanent_uuid: "dcbc18964e3d49118bb745a1f89bd76f" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 43771 } } peers { permanent_uuid: "b8b4c95e9c12440eada45ae22e226f9f" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 42875 } }
I20260807 23:01:32.900071 24685 raft_consensus.cc:374] T 09af4f448d314336ac58b26e8f862c68 P dcbc18964e3d49118bb745a1f89bd76f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 23:01:32.900198 24685 raft_consensus.cc:729] T 09af4f448d314336ac58b26e8f862c68 P dcbc18964e3d49118bb745a1f89bd76f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcbc18964e3d49118bb745a1f89bd76f, State: Initialized, Role: FOLLOWER
I20260807 23:01:32.900893 24685 consensus_queue.cc:260] T 09af4f448d314336ac58b26e8f862c68 P dcbc18964e3d49118bb745a1f89bd76f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "484ca854ed3e44f4a26e1fcbee681427" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 43709 } } peers { permanent_uuid: "dcbc18964e3d49118bb745a1f89bd76f" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 43771 } } peers { permanent_uuid: "b8b4c95e9c12440eada45ae22e226f9f" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 42875 } }
I20260807 23:01:32.902143 24656 heartbeater.cc:503] Master 127.25.94.62:44451 was elected leader, sending a full tablet report...
I20260807 23:01:32.902951 24685 ts_tablet_manager.cc:1428] T 09af4f448d314336ac58b26e8f862c68 P dcbc18964e3d49118bb745a1f89bd76f: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
W20260807 23:01:32.935287 24459 tablet.cc:2369] T 09af4f448d314336ac58b26e8f862c68 P 484ca854ed3e44f4a26e1fcbee681427: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 23:01:33.016928 24657 tablet.cc:2369] T 09af4f448d314336ac58b26e8f862c68 P dcbc18964e3d49118bb745a1f89bd76f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 23:01:33.019886 24693 raft_consensus.cc:482] T 09af4f448d314336ac58b26e8f862c68 P 484ca854ed3e44f4a26e1fcbee681427 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 23:01:33.020260 24693 raft_consensus.cc:504] T 09af4f448d314336ac58b26e8f862c68 P 484ca854ed3e44f4a26e1fcbee681427 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "484ca854ed3e44f4a26e1fcbee681427" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 43709 } } peers { permanent_uuid: "dcbc18964e3d49118bb745a1f89bd76f" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 43771 } } peers { permanent_uuid: "b8b4c95e9c12440eada45ae22e226f9f" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 42875 } }
I20260807 23:01:33.022015 24693 leader_election.cc:288] T 09af4f448d314336ac58b26e8f862c68 P 484ca854ed3e44f4a26e1fcbee681427 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dcbc18964e3d49118bb745a1f89bd76f (127.25.94.3:43771), b8b4c95e9c12440eada45ae22e226f9f (127.25.94.1:42875)
I20260807 23:01:33.031795 24678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:33.022555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.2:46281 (local address 127.25.94.1:42875)
0807 23:01:33.022748 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0807 23:01:33.022756 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 23:01:33.023578 (+ 822us) server_negotiation.cc:407] Connection header received
0807 23:01:33.023694 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:01:33.023699 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:01:33.023780 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:01:33.023887 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 23:01:33.024682 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:33.025861 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:33.029011 (+ 3150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:33.030365 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:33.030514 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:33.030640 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0807 23:01:33.031366 (+ 726us) server_negotiation.cc:299] Negotiation successful
0807 23:01:33.031525 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260807 23:01:33.031811 24674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:33.022459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.2:34265 (local address 127.25.94.3:43771)
0807 23:01:33.022667 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0807 23:01:33.022673 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 23:01:33.023499 (+ 826us) server_negotiation.cc:407] Connection header received
0807 23:01:33.023701 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:01:33.023706 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:01:33.023780 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:01:33.023888 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 23:01:33.024682 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:33.025861 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:33.029013 (+ 3152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:33.030365 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:33.030514 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:33.030640 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0807 23:01:33.031366 (+ 726us) server_negotiation.cc:299] Negotiation successful
0807 23:01:33.031525 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260807 23:01:33.032029 24698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:33.022322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.3:43771 (local address 127.25.94.2:34265)
0807 23:01:33.023353 (+ 1031us) negotiation.cc:107] Waiting for socket to connect
0807 23:01:33.023376 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0807 23:01:33.023552 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 23:01:33.023922 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 23:01:33.023927 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 23:01:33.024016 (+ 89us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 23:01:33.024516 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:01:33.024529 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:01:33.026023 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:01:33.026029 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:01:33.028790 (+ 2761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:01:33.028808 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:01:33.030716 (+ 1908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:01:33.030721 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:01:33.031028 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:33.031167 (+ 139us) client_negotiation.cc:769] Sending connection context
0807 23:01:33.031278 (+ 111us) client_negotiation.cc:240] Negotiation successful
0807 23:01:33.031463 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":842,"mutex_wait_us":256,"spinlock_wait_cycles":22272,"thread_start_us":685,"threads_started":1}
I20260807 23:01:33.032409 24697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:33.022444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.1:42875 (local address 127.25.94.2:46281)
0807 23:01:33.023453 (+ 1009us) negotiation.cc:107] Waiting for socket to connect
0807 23:01:33.023469 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 23:01:33.023563 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 23:01:33.023922 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 23:01:33.023927 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 23:01:33.023953 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 23:01:33.024474 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:01:33.024493 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:01:33.026023 (+ 1530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:01:33.026029 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:01:33.028795 (+ 2766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:01:33.028808 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:01:33.030708 (+ 1900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:01:33.030715 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:01:33.031028 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:33.031168 (+ 140us) client_negotiation.cc:769] Sending connection context
0807 23:01:33.031278 (+ 110us) client_negotiation.cc:240] Negotiation successful
0807 23:01:33.031463 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":890,"mutex_wait_us":292,"thread_start_us":431,"threads_started":1}
I20260807 23:01:33.033149 24585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09af4f448d314336ac58b26e8f862c68" candidate_uuid: "484ca854ed3e44f4a26e1fcbee681427" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcbc18964e3d49118bb745a1f89bd76f" is_pre_election: true
I20260807 23:01:33.033561 24178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09af4f448d314336ac58b26e8f862c68" candidate_uuid: "484ca854ed3e44f4a26e1fcbee681427" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8b4c95e9c12440eada45ae22e226f9f" is_pre_election: true
I20260807 23:01:33.033829 24585 raft_consensus.cc:2454] T 09af4f448d314336ac58b26e8f862c68 P dcbc18964e3d49118bb745a1f89bd76f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 484ca854ed3e44f4a26e1fcbee681427 in term 0.
I20260807 23:01:33.034147 24178 raft_consensus.cc:2454] T 09af4f448d314336ac58b26e8f862c68 P b8b4c95e9c12440eada45ae22e226f9f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 484ca854ed3e44f4a26e1fcbee681427 in term 0.
I20260807 23:01:33.034613 24284 leader_election.cc:302] T 09af4f448d314336ac58b26e8f862c68 P 484ca854ed3e44f4a26e1fcbee681427 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 484ca854ed3e44f4a26e1fcbee681427, dcbc18964e3d49118bb745a1f89bd76f; no voters:
I20260807 23:01:33.035113 24693 raft_consensus.cc:2781] T 09af4f448d314336ac58b26e8f862c68 P 484ca854ed3e44f4a26e1fcbee681427 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 23:01:33.035285 24693 raft_consensus.cc:482] T 09af4f448d314336ac58b26e8f862c68 P 484ca854ed3e44f4a26e1fcbee681427 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 23:01:33.035429 24693 raft_consensus.cc:3037] T 09af4f448d314336ac58b26e8f862c68 P 484ca854ed3e44f4a26e1fcbee681427 [term 0 FOLLOWER]: Advancing to term 1
I20260807 23:01:33.037740 24693 raft_consensus.cc:504] T 09af4f448d314336ac58b26e8f862c68 P 484ca854ed3e44f4a26e1fcbee681427 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "484ca854ed3e44f4a26e1fcbee681427" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 43709 } } peers { permanent_uuid: "dcbc18964e3d49118bb745a1f89bd76f" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 43771 } } peers { permanent_uuid: "b8b4c95e9c12440eada45ae22e226f9f" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 42875 } }
I20260807 23:01:33.038394 24693 leader_election.cc:288] T 09af4f448d314336ac58b26e8f862c68 P 484ca854ed3e44f4a26e1fcbee681427 [CANDIDATE]: Term 1 election: Requested vote from peers dcbc18964e3d49118bb745a1f89bd76f (127.25.94.3:43771), b8b4c95e9c12440eada45ae22e226f9f (127.25.94.1:42875)
I20260807 23:01:33.038887 24178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09af4f448d314336ac58b26e8f862c68" candidate_uuid: "484ca854ed3e44f4a26e1fcbee681427" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8b4c95e9c12440eada45ae22e226f9f"
I20260807 23:01:33.038910 24585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09af4f448d314336ac58b26e8f862c68" candidate_uuid: "484ca854ed3e44f4a26e1fcbee681427" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcbc18964e3d49118bb745a1f89bd76f"
I20260807 23:01:33.039103 24178 raft_consensus.cc:3037] T 09af4f448d314336ac58b26e8f862c68 P b8b4c95e9c12440eada45ae22e226f9f [term 0 FOLLOWER]: Advancing to term 1
I20260807 23:01:33.039105 24585 raft_consensus.cc:3037] T 09af4f448d314336ac58b26e8f862c68 P dcbc18964e3d49118bb745a1f89bd76f [term 0 FOLLOWER]: Advancing to term 1
I20260807 23:01:33.041406 24585 raft_consensus.cc:2454] T 09af4f448d314336ac58b26e8f862c68 P dcbc18964e3d49118bb745a1f89bd76f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 484ca854ed3e44f4a26e1fcbee681427 in term 1.
I20260807 23:01:33.041407 24178 raft_consensus.cc:2454] T 09af4f448d314336ac58b26e8f862c68 P b8b4c95e9c12440eada45ae22e226f9f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 484ca854ed3e44f4a26e1fcbee681427 in term 1.
I20260807 23:01:33.041846 24284 leader_election.cc:302] T 09af4f448d314336ac58b26e8f862c68 P 484ca854ed3e44f4a26e1fcbee681427 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 484ca854ed3e44f4a26e1fcbee681427, dcbc18964e3d49118bb745a1f89bd76f; no voters:
I20260807 23:01:33.042160 24693 raft_consensus.cc:2781] T 09af4f448d314336ac58b26e8f862c68 P 484ca854ed3e44f4a26e1fcbee681427 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 23:01:33.042626 24693 raft_consensus.cc:686] T 09af4f448d314336ac58b26e8f862c68 P 484ca854ed3e44f4a26e1fcbee681427 [term 1 LEADER]: Becoming Leader. State: Replica: 484ca854ed3e44f4a26e1fcbee681427, State: Running, Role: LEADER
I20260807 23:01:33.043111 24693 consensus_queue.cc:237] T 09af4f448d314336ac58b26e8f862c68 P 484ca854ed3e44f4a26e1fcbee681427 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "484ca854ed3e44f4a26e1fcbee681427" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 43709 } } peers { permanent_uuid: "dcbc18964e3d49118bb745a1f89bd76f" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 43771 } } peers { permanent_uuid: "b8b4c95e9c12440eada45ae22e226f9f" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 42875 } }
I20260807 23:01:33.047935 23934 catalog_manager.cc:5495] T 09af4f448d314336ac58b26e8f862c68 P 484ca854ed3e44f4a26e1fcbee681427 reported cstate change: term changed from 0 to 1, leader changed from <none> to 484ca854ed3e44f4a26e1fcbee681427 (127.25.94.2). New cstate: current_term: 1 leader_uuid: "484ca854ed3e44f4a26e1fcbee681427" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "484ca854ed3e44f4a26e1fcbee681427" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 43709 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dcbc18964e3d49118bb745a1f89bd76f" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 43771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b8b4c95e9c12440eada45ae22e226f9f" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 42875 } health_report { overall_health: UNKNOWN } } }
I20260807 23:01:33.094480 24673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:33.088611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50110 (local address 127.25.94.2:43709)
0807 23:01:33.088831 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0807 23:01:33.088836 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 23:01:33.088871 (+ 35us) server_negotiation.cc:407] Connection header received
0807 23:01:33.088957 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:01:33.088961 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:01:33.089045 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:01:33.089276 (+ 231us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 23:01:33.089606 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:33.090352 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:33.091867 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:33.092313 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:33.092413 (+ 100us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:33.093049 (+ 636us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 23:01:33.093654 (+ 605us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 23:01:33.093815 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0807 23:01:33.094009 (+ 194us) server_negotiation.cc:299] Negotiation successful
0807 23:01:33.094081 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260807 23:01:33.104038 24585 raft_consensus.cc:1261] T 09af4f448d314336ac58b26e8f862c68 P dcbc18964e3d49118bb745a1f89bd76f [term 1 FOLLOWER]: Refusing update from remote peer 484ca854ed3e44f4a26e1fcbee681427: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 23:01:33.104105 24178 raft_consensus.cc:1261] T 09af4f448d314336ac58b26e8f862c68 P b8b4c95e9c12440eada45ae22e226f9f [term 1 FOLLOWER]: Refusing update from remote peer 484ca854ed3e44f4a26e1fcbee681427: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 23:01:33.105095 24693 consensus_queue.cc:1035] T 09af4f448d314336ac58b26e8f862c68 P 484ca854ed3e44f4a26e1fcbee681427 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8b4c95e9c12440eada45ae22e226f9f" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 42875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 23:01:33.105491 24699 consensus_queue.cc:1035] T 09af4f448d314336ac58b26e8f862c68 P 484ca854ed3e44f4a26e1fcbee681427 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dcbc18964e3d49118bb745a1f89bd76f" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 43771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 23:01:33.134104 23936 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52370:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 23:01:33.134441 23936 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52370:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 23:01:33.138883 23936 catalog_manager.cc:5782] T 00000000000000000000000000000000 P eaa76a92ba624f99a472d502538706b4: Sending DeleteTablet for 3 replicas of tablet 09af4f448d314336ac58b26e8f862c68
I20260807 23:01:33.140275 24352 tablet_service.cc:1514] Processing DeleteTablet for tablet 09af4f448d314336ac58b26e8f862c68 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 23:01:33 UTC) from {username='kudu', principal='kudu/127.25.94.62@KRBTEST.COM'} at 127.0.0.1:50098
I20260807 23:01:33.140513 24143 tablet_service.cc:1514] Processing DeleteTablet for tablet 09af4f448d314336ac58b26e8f862c68 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 23:01:33 UTC) from {username='kudu', principal='kudu/127.25.94.62@KRBTEST.COM'} at 127.0.0.1:38060
I20260807 23:01:33.140990 24145 tablet_service.cc:1514] Processing DeleteTablet for tablet 09af4f448d314336ac58b26e8f862c68 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 23:01:33 UTC) from {username='kudu', principal='kudu/127.25.94.62@KRBTEST.COM'} at 127.0.0.1:38060
I20260807 23:01:33.141042 24557 tablet_service.cc:1514] Processing DeleteTablet for tablet 09af4f448d314336ac58b26e8f862c68 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 23:01:33 UTC) from {username='kudu', principal='kudu/127.25.94.62@KRBTEST.COM'} at 127.0.0.1:38288
I20260807 23:01:33.143651 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 24042
I20260807 23:01:33.152494 24724 tablet_replica.cc:330] T 09af4f448d314336ac58b26e8f862c68 P dcbc18964e3d49118bb745a1f89bd76f: stopping tablet replica
I20260807 23:01:33.153213 24724 raft_consensus.cc:2229] T 09af4f448d314336ac58b26e8f862c68 P dcbc18964e3d49118bb745a1f89bd76f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 23:01:33.154001 24721 tablet_replica.cc:330] T 09af4f448d314336ac58b26e8f862c68 P 484ca854ed3e44f4a26e1fcbee681427: stopping tablet replica
I20260807 23:01:33.154001 24724 raft_consensus.cc:2258] T 09af4f448d314336ac58b26e8f862c68 P dcbc18964e3d49118bb745a1f89bd76f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 23:01:33.154481 24721 raft_consensus.cc:2229] T 09af4f448d314336ac58b26e8f862c68 P 484ca854ed3e44f4a26e1fcbee681427 [term 1 LEADER]: Raft consensus shutting down.
I20260807 23:01:33.155802 24723 tablet_replica.cc:330] T 09af4f448d314336ac58b26e8f862c68 P b8b4c95e9c12440eada45ae22e226f9f: stopping tablet replica
I20260807 23:01:33.156229 24723 raft_consensus.cc:2229] T 09af4f448d314336ac58b26e8f862c68 P b8b4c95e9c12440eada45ae22e226f9f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 23:01:33.156764 24723 raft_consensus.cc:2258] T 09af4f448d314336ac58b26e8f862c68 P b8b4c95e9c12440eada45ae22e226f9f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 23:01:33.156792 24724 ts_tablet_manager.cc:1905] T 09af4f448d314336ac58b26e8f862c68 P dcbc18964e3d49118bb745a1f89bd76f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 23:01:33.157261 24721 raft_consensus.cc:2258] T 09af4f448d314336ac58b26e8f862c68 P 484ca854ed3e44f4a26e1fcbee681427 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 23:01:33.159372 24723 ts_tablet_manager.cc:1905] T 09af4f448d314336ac58b26e8f862c68 P b8b4c95e9c12440eada45ae22e226f9f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 23:01:33.164608 24724 ts_tablet_manager.cc:1918] T 09af4f448d314336ac58b26e8f862c68 P dcbc18964e3d49118bb745a1f89bd76f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 23:01:33.164824 24724 log.cc:1196] T 09af4f448d314336ac58b26e8f862c68 P dcbc18964e3d49118bb745a1f89bd76f: Deleting WAL directory at /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786143361779964-25976-0/minicluster-data/ts-2/wal/wals/09af4f448d314336ac58b26e8f862c68
I20260807 23:01:33.165685 24724 ts_tablet_manager.cc:1939] T 09af4f448d314336ac58b26e8f862c68 P dcbc18964e3d49118bb745a1f89bd76f: Deleting consensus metadata
I20260807 23:01:33.167412 24723 ts_tablet_manager.cc:1918] T 09af4f448d314336ac58b26e8f862c68 P b8b4c95e9c12440eada45ae22e226f9f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 23:01:33.167606 24723 log.cc:1196] T 09af4f448d314336ac58b26e8f862c68 P b8b4c95e9c12440eada45ae22e226f9f: Deleting WAL directory at /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786143361779964-25976-0/minicluster-data/ts-0/wal/wals/09af4f448d314336ac58b26e8f862c68
I20260807 23:01:33.168102 23915 catalog_manager.cc:4841] TS dcbc18964e3d49118bb745a1f89bd76f (127.25.94.3:43771): tablet 09af4f448d314336ac58b26e8f862c68 (table test-table [id=1424efd7c25f4fd5985586e450ee6b47]) successfully deleted
I20260807 23:01:33.168231 24723 ts_tablet_manager.cc:1939] T 09af4f448d314336ac58b26e8f862c68 P b8b4c95e9c12440eada45ae22e226f9f: Deleting consensus metadata
I20260807 23:01:33.169912 24721 ts_tablet_manager.cc:1905] T 09af4f448d314336ac58b26e8f862c68 P 484ca854ed3e44f4a26e1fcbee681427: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 23:01:33.170282 23911 catalog_manager.cc:4841] TS b8b4c95e9c12440eada45ae22e226f9f (127.25.94.1:42875): tablet 09af4f448d314336ac58b26e8f862c68 (table test-table [id=1424efd7c25f4fd5985586e450ee6b47]) successfully deleted
W20260807 23:01:33.171308 23911 catalog_manager.cc:4808] TS b8b4c95e9c12440eada45ae22e226f9f (127.25.94.1:42875): delete failed for tablet 09af4f448d314336ac58b26e8f862c68 because the tablet was not found. No further retry: Not found: Tablet not found: 09af4f448d314336ac58b26e8f862c68
Aug 07 23:01:33 dist-test-slave-z8x9 krb5kdc[23870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143690, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.1@KRBTEST.COM
I20260807 23:01:33.176481 24721 ts_tablet_manager.cc:1918] T 09af4f448d314336ac58b26e8f862c68 P 484ca854ed3e44f4a26e1fcbee681427: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 23:01:33.176707 24721 log.cc:1196] T 09af4f448d314336ac58b26e8f862c68 P 484ca854ed3e44f4a26e1fcbee681427: Deleting WAL directory at /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786143361779964-25976-0/minicluster-data/ts-1/wal/wals/09af4f448d314336ac58b26e8f862c68
I20260807 23:01:33.177347 24721 ts_tablet_manager.cc:1939] T 09af4f448d314336ac58b26e8f862c68 P 484ca854ed3e44f4a26e1fcbee681427: Deleting consensus metadata
I20260807 23:01:33.179275 23911 catalog_manager.cc:4841] TS 484ca854ed3e44f4a26e1fcbee681427 (127.25.94.2:43709): tablet 09af4f448d314336ac58b26e8f862c68 (table test-table [id=1424efd7c25f4fd5985586e450ee6b47]) successfully deleted
I20260807 23:01:33.186314 24678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:33.147243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38072 (local address 127.25.94.1:42875)
0807 23:01:33.147484 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0807 23:01:33.147493 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 23:01:33.149275 (+ 1782us) server_negotiation.cc:407] Connection header received
0807 23:01:33.149814 (+ 539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:01:33.149824 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:01:33.149994 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:01:33.150132 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:01:33.163736 (+ 13604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:33.165395 (+ 1659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:33.168089 (+ 2694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:33.168958 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:33.169083 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:33.177282 (+ 8199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:01:33.177305 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:01:33.177326 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 23:01:33.177384 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:01:33.181953 (+ 4569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:01:33.183063 (+ 1110us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:01:33.183076 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:01:33.183082 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:01:33.183273 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:01:33.183840 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:01:33.183846 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:01:33.183858 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:01:33.184401 (+ 543us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:01:33.184711 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0807 23:01:33.185336 (+ 625us) server_negotiation.cc:299] Negotiation successful
0807 23:01:33.185615 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260807 23:01:33.187078 24242 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 23:01:33.287264 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 24042
I20260807 23:01:33.313081 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 24255
Aug 07 23:01:33 dist-test-slave-z8x9 krb5kdc[23870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143690, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.2@KRBTEST.COM
I20260807 23:01:33.332449 24673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:33.314243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50126 (local address 127.25.94.2:43709)
0807 23:01:33.314462 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0807 23:01:33.314468 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 23:01:33.314490 (+ 22us) server_negotiation.cc:407] Connection header received
0807 23:01:33.314591 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:01:33.314597 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:01:33.314676 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:01:33.314795 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:01:33.316241 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:33.317598 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:33.318924 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:33.319617 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:33.319755 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:33.324709 (+ 4954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:01:33.324732 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:01:33.324759 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 23:01:33.324815 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:01:33.329197 (+ 4382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:01:33.330035 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:01:33.330045 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:01:33.330048 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:01:33.330229 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:01:33.330614 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:01:33.330620 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:01:33.330624 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:01:33.331107 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:01:33.331488 (+ 381us) server_negotiation.cc:1035] Waiting for connection context
0807 23:01:33.331771 (+ 283us) server_negotiation.cc:299] Negotiation successful
0807 23:01:33.332028 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260807 23:01:33.333415 24447 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 23:01:33.450163 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 24255
I20260807 23:01:33.473663 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 24465
Aug 07 23:01:33 dist-test-slave-z8x9 krb5kdc[23870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143690, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.3@KRBTEST.COM
I20260807 23:01:33.493525 24674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:33.474448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38296 (local address 127.25.94.3:43771)
0807 23:01:33.474647 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0807 23:01:33.474652 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 23:01:33.474671 (+ 19us) server_negotiation.cc:407] Connection header received
0807 23:01:33.474842 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:01:33.474846 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:01:33.474910 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:01:33.474996 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:01:33.476323 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:33.477060 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:33.478370 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:33.478749 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:33.478846 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:33.488153 (+ 9307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:01:33.488168 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:01:33.488185 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 23:01:33.488221 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:01:33.490953 (+ 2732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:01:33.491680 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:01:33.491689 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:01:33.491691 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:01:33.491832 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:01:33.492165 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:01:33.492168 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:01:33.492170 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:01:33.492512 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:01:33.492729 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0807 23:01:33.492997 (+ 268us) server_negotiation.cc:299] Negotiation successful
0807 23:01:33.493159 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260807 23:01:33.494161 24650 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 23:01:33.584873 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 24465
I20260807 23:01:33.608896 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 23889
I20260807 23:01:33.610092 23982 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 23:01:33.721666 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 23889
2026-08-07T23:01:33Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4908 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-07T23:01:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T23:01:33Z Disabled control of system clock
I20260807 23:01:33.812534 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786143361779964-25976-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786143361779964-25976-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786143361779964-25976-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.62:36359
--webserver_interface=127.25.94.62
--webserver_port=0
--builtin_ntp_servers=127.25.94.20:33073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.94.62:36359
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260807 23:01:34.159955 24742 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 23:01:34.160287 24742 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 23:01:34.160395 24742 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 23:01:34.168560 24742 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 23:01:34.168701 24742 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 23:01:34.168741 24742 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 23:01:34.168793 24742 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 23:01:34.168838 24742 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 23:01:34.178737 24742 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:33073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786143361779964-25976-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786143361779964-25976-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.94.62:36359
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.62:36359
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.94.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786143361779964-25976-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 23:01:34.180115 24742 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 23:01:34.182144 24742 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 23:01:34.195006 24750 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 23:01:34.195051 24752 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 23:01:34.195080 24754 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 23:01:34.196110 24742 server_base.cc:1034] running on GCE node
I20260807 23:01:34.197009 24742 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 23:01:34.198612 24742 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 23:01:34.199887 24742 hybrid_clock.cc:648] HybridClock initialized: now 1786143694199850 us; error 64 us; skew 500 ppm
I20260807 23:01:34.200330 24742 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 23:01:34.203154 24742 webserver.cc:460] Webserver started at http://127.25.94.62:36363/ using document root <none> and password file <none>
I20260807 23:01:34.203960 24742 fs_manager.cc:359] Metadata directory not provided
I20260807 23:01:34.204110 24742 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 23:01:34.204484 24742 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 23:01:34.207695 24742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786143361779964-25976-0/minicluster-data/master-0/data/instance:
uuid: "a3fd7634127a4553937a163952df0d7b"
format_stamp: "Formatted at 2026-08-07 23:01:34 on dist-test-slave-z8x9"
server_key: "46f09b2b88b94ec7fe7fbfa05701933e"
server_key_iv: "f82354096c54896c16cf296e2a4e3805"
server_key_version: "encryptionkey@0"
I20260807 23:01:34.208618 24742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786143361779964-25976-0/minicluster-data/master-0/wal/instance:
uuid: "a3fd7634127a4553937a163952df0d7b"
format_stamp: "Formatted at 2026-08-07 23:01:34 on dist-test-slave-z8x9"
server_key: "46f09b2b88b94ec7fe7fbfa05701933e"
server_key_iv: "f82354096c54896c16cf296e2a4e3805"
server_key_version: "encryptionkey@0"
I20260807 23:01:34.214373 24742 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260807 23:01:34.218510 24766 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 23:01:34.220144 24742 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20260807 23:01:34.220404 24742 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786143361779964-25976-0/minicluster-data/master-0/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786143361779964-25976-0/minicluster-data/master-0/wal
uuid: "a3fd7634127a4553937a163952df0d7b"
format_stamp: "Formatted at 2026-08-07 23:01:34 on dist-test-slave-z8x9"
server_key: "46f09b2b88b94ec7fe7fbfa05701933e"
server_key_iv: "f82354096c54896c16cf296e2a4e3805"
server_key_version: "encryptionkey@0"
I20260807 23:01:34.220664 24742 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786143361779964-25976-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786143361779964-25976-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786143361779964-25976-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 23:01:34.238036 24742 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 23:01:34.238977 24742 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 23:01:34.239308 24742 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 23:01:34.259538 24742 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.62:36359
I20260807 23:01:34.259549 24834 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.62:36359 every 8 connection(s)
I20260807 23:01:34.260969 24742 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
I20260807 23:01:34.263605 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 24742
I20260807 23:01:34.263816 25976 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786143361779964-25976-0/minicluster-data/master-0/wal/instance
I20260807 23:01:34.264495 25976 external_mini_cluster.cc:1428] Setting key 6cdab101a29364edd455958a7d2bb914
I20260807 23:01:34.265318 24835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 23:01:34.279278 24835 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3fd7634127a4553937a163952df0d7b: Bootstrap starting.
I20260807 23:01:34.280501 24840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:34.266646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42304 (local address 127.25.94.62:36359)
0807 23:01:34.267808 (+ 1162us) server_negotiation.cc:206] Beginning negotiation
0807 23:01:34.267826 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0807 23:01:34.267962 (+ 136us) server_negotiation.cc:407] Connection header received
0807 23:01:34.270706 (+ 2744us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:01:34.270732 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:01:34.271260 (+ 528us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:01:34.271694 (+ 434us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:01:34.272325 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:34.273766 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:34.275146 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:34.275859 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:34.276078 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:34.277532 (+ 1454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:01:34.277540 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:01:34.277556 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 23:01:34.278289 (+ 733us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:01:34.278599 (+ 310us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 23:01:34.278627 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:01:34.278801 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0807 23:01:34.279080 (+ 279us) server_negotiation.cc:299] Negotiation successful
0807 23:01:34.279456 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":619,"thread_start_us":407,"threads_started":1}
I20260807 23:01:34.282793 24835 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a3fd7634127a4553937a163952df0d7b: Neither blocks nor log segments found. Creating new log.
I20260807 23:01:34.284009 24835 log.cc:824] T 00000000000000000000000000000000 P a3fd7634127a4553937a163952df0d7b: Log is configured to *not* fsync() on all Append() calls
I20260807 23:01:34.286957 24835 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3fd7634127a4553937a163952df0d7b: No bootstrap required, opened a new log
I20260807 23:01:34.295285 24835 raft_consensus.cc:348] T 00000000000000000000000000000000 P a3fd7634127a4553937a163952df0d7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3fd7634127a4553937a163952df0d7b" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 36359 } }
I20260807 23:01:34.295729 24835 raft_consensus.cc:374] T 00000000000000000000000000000000 P a3fd7634127a4553937a163952df0d7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 23:01:34.295857 24835 raft_consensus.cc:729] T 00000000000000000000000000000000 P a3fd7634127a4553937a163952df0d7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3fd7634127a4553937a163952df0d7b, State: Initialized, Role: FOLLOWER
I20260807 23:01:34.296587 24835 consensus_queue.cc:260] T 00000000000000000000000000000000 P a3fd7634127a4553937a163952df0d7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3fd7634127a4553937a163952df0d7b" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 36359 } }
I20260807 23:01:34.296877 24835 raft_consensus.cc:388] T 00000000000000000000000000000000 P a3fd7634127a4553937a163952df0d7b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 23:01:34.297011 24835 raft_consensus.cc:482] T 00000000000000000000000000000000 P a3fd7634127a4553937a163952df0d7b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 23:01:34.297219 24835 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a3fd7634127a4553937a163952df0d7b [term 0 FOLLOWER]: Advancing to term 1
I20260807 23:01:34.299432 24835 raft_consensus.cc:504] T 00000000000000000000000000000000 P a3fd7634127a4553937a163952df0d7b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3fd7634127a4553937a163952df0d7b" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 36359 } }
I20260807 23:01:34.299937 24835 leader_election.cc:302] T 00000000000000000000000000000000 P a3fd7634127a4553937a163952df0d7b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a3fd7634127a4553937a163952df0d7b; no voters:
I20260807 23:01:34.300560 24835 leader_election.cc:288] T 00000000000000000000000000000000 P a3fd7634127a4553937a163952df0d7b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 23:01:34.300751 24843 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a3fd7634127a4553937a163952df0d7b [term 1 FOLLOWER]: Leader election won for term 1
I20260807 23:01:34.301934 24843 raft_consensus.cc:686] T 00000000000000000000000000000000 P a3fd7634127a4553937a163952df0d7b [term 1 LEADER]: Becoming Leader. State: Replica: a3fd7634127a4553937a163952df0d7b, State: Running, Role: LEADER
I20260807 23:01:34.302531 24843 consensus_queue.cc:237] T 00000000000000000000000000000000 P a3fd7634127a4553937a163952df0d7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3fd7634127a4553937a163952df0d7b" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 36359 } }
I20260807 23:01:34.303117 24835 sys_catalog.cc:564] T 00000000000000000000000000000000 P a3fd7634127a4553937a163952df0d7b [sys.catalog]: configured and running, proceeding with master startup.
I20260807 23:01:34.305680 24847 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3fd7634127a4553937a163952df0d7b [sys.catalog]: SysCatalogTable state changed. Reason: New leader a3fd7634127a4553937a163952df0d7b. Latest consensus state: current_term: 1 leader_uuid: "a3fd7634127a4553937a163952df0d7b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3fd7634127a4553937a163952df0d7b" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 36359 } } }
I20260807 23:01:34.305514 24845 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3fd7634127a4553937a163952df0d7b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a3fd7634127a4553937a163952df0d7b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3fd7634127a4553937a163952df0d7b" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 36359 } } }
I20260807 23:01:34.306048 24847 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3fd7634127a4553937a163952df0d7b [sys.catalog]: This master's current role is: LEADER
I20260807 23:01:34.306164 24845 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3fd7634127a4553937a163952df0d7b [sys.catalog]: This master's current role is: LEADER
I20260807 23:01:34.307294 24856 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 23:01:34.316910 24856 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 23:01:34.327507 24856 catalog_manager.cc:1347] Generated new cluster ID: 798f679c455041af9a7f46e24f861630
I20260807 23:01:34.327674 24856 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 23:01:34.344743 24856 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 23:01:34.346354 24856 catalog_manager.cc:1504] Loading token signing keys...
I20260807 23:01:34.359579 24856 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a3fd7634127a4553937a163952df0d7b: Generated new TSK 0
I20260807 23:01:34.360828 24856 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 23:01:34.379830 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786143361779964-25976-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786143361779964-25976-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786143361779964-25976-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786143361779964-25976-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.1:0
--local_ip_for_outbound_sockets=127.25.94.1
--webserver_interface=127.25.94.1
--webserver_port=0
--tserver_master_addrs=127.25.94.62:36359
--builtin_ntp_servers=127.25.94.20:33073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260807 23:01:34.737960 24876 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 23:01:34.738252 24876 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 23:01:34.738349 24876 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 23:01:34.745690 24876 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 23:01:34.745798 24876 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 23:01:34.745903 24876 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.1
I20260807 23:01:34.756405 24876 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:33073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786143361779964-25976-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786143361779964-25976-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786143361779964-25976-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.94.1
--webserver_port=0
--tserver_master_addrs=127.25.94.62:36359
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.1
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786143361779964-25976-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 23:01:34.757750 24876 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 23:01:34.759498 24876 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 23:01:34.773622 24884 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 23:01:34.773646 24883 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 23:01:34.773958 24886 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 23:01:34.774636 24876 server_base.cc:1034] running on GCE node
I20260807 23:01:34.775281 24876 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 23:01:34.776291 24876 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 23:01:34.777493 24876 hybrid_clock.cc:648] HybridClock initialized: now 1786143694777433 us; error 52 us; skew 500 ppm
I20260807 23:01:34.777822 24876 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 23:01:34.780128 24876 webserver.cc:460] Webserver started at http://127.25.94.1:36459/ using document root <none> and password file <none>
I20260807 23:01:34.780897 24876 fs_manager.cc:359] Metadata directory not provided
I20260807 23:01:34.781013 24876 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 23:01:34.781415 24876 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 23:01:34.784274 24876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786143361779964-25976-0/minicluster-data/ts-0/data/instance:
uuid: "91c7e3c5fe8d4c9c90622ec014280afd"
format_stamp: "Formatted at 2026-08-07 23:01:34 on dist-test-slave-z8x9"
server_key: "f0dc89cd2dcda74b1d3a3dbbd4aa92ff"
server_key_iv: "8f74e8ab4fb52e0c95a3602b2b2cb892"
server_key_version: "encryptionkey@0"
I20260807 23:01:34.785073 24876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786143361779964-25976-0/minicluster-data/ts-0/wal/instance:
uuid: "91c7e3c5fe8d4c9c90622ec014280afd"
format_stamp: "Formatted at 2026-08-07 23:01:34 on dist-test-slave-z8x9"
server_key: "f0dc89cd2dcda74b1d3a3dbbd4aa92ff"
server_key_iv: "8f74e8ab4fb52e0c95a3602b2b2cb892"
server_key_version: "encryptionkey@0"
I20260807 23:01:34.790513 24876 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260807 23:01:34.794377 24896 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 23:01:34.795723 24876 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260807 23:01:34.795903 24876 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786143361779964-25976-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786143361779964-25976-0/minicluster-data/ts-0/wal
uuid: "91c7e3c5fe8d4c9c90622ec014280afd"
format_stamp: "Formatted at 2026-08-07 23:01:34 on dist-test-slave-z8x9"
server_key: "f0dc89cd2dcda74b1d3a3dbbd4aa92ff"
server_key_iv: "8f74e8ab4fb52e0c95a3602b2b2cb892"
server_key_version: "encryptionkey@0"
I20260807 23:01:34.796140 24876 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786143361779964-25976-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786143361779964-25976-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786143361779964-25976-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 23:01:34.815212 24876 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 23:01:34.816054 24876 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 23:01:34.816346 24876 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 23:01:34.817492 24876 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 23:01:34.819000 24876 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 23:01:34.819130 24876 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 23:01:34.819263 24876 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 23:01:34.819378 24876 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 23:01:34.853399 24876 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.1:44931
I20260807 23:01:34.853492 25090 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.1:44931 every 8 connection(s)
I20260807 23:01:34.854808 24876 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786143361779964-25976-0/minicluster-data/ts-0/data/info.pb
I20260807 23:01:34.862823 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 24876
I20260807 23:01:34.863034 25976 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786143361779964-25976-0/minicluster-data/ts-0/wal/instance
I20260807 23:01:34.863512 25976 external_mini_cluster.cc:1428] Setting key daf6a3e707e78d6137101791fe80b8d5
I20260807 23:01:34.866539 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786143361779964-25976-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786143361779964-25976-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786143361779964-25976-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786143361779964-25976-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.2:0
--local_ip_for_outbound_sockets=127.25.94.2
--webserver_interface=127.25.94.2
--webserver_port=0
--tserver_master_addrs=127.25.94.62:36359
--builtin_ntp_servers=127.25.94.20:33073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260807 23:01:34.867866 25094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:34.857333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.1:55255 (local address 127.25.94.62:36359)
0807 23:01:34.857770 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0807 23:01:34.857774 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 23:01:34.858867 (+ 1093us) server_negotiation.cc:407] Connection header received
0807 23:01:34.860641 (+ 1774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:01:34.860646 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:01:34.860739 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:01:34.860859 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:01:34.861994 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:34.862782 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:34.864580 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:34.865160 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:34.865407 (+ 247us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:34.866969 (+ 1562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:01:34.866975 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:01:34.866978 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 23:01:34.866996 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:01:34.867035 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 23:01:34.867047 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:01:34.867147 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0807 23:01:34.867382 (+ 235us) server_negotiation.cc:299] Negotiation successful
0807 23:01:34.867479 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":235,"threads_started":1}
I20260807 23:01:34.868854 25095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:34.857814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:36359 (local address 127.25.94.1:55255)
0807 23:01:34.858704 (+ 890us) negotiation.cc:107] Waiting for socket to connect
0807 23:01:34.858745 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0807 23:01:34.860376 (+ 1631us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 23:01:34.861112 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 23:01:34.861120 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 23:01:34.861260 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0807 23:01:34.861823 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:01:34.861840 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:01:34.862949 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:01:34.862975 (+ 26us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:01:34.864416 (+ 1441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:01:34.864430 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:01:34.865498 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:01:34.865506 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:01:34.865745 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:34.866547 (+ 802us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 23:01:34.866550 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0807 23:01:34.866590 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 23:01:34.866721 (+ 131us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 23:01:34.867149 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 23:01:34.867163 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 23:01:34.867173 (+ 10us) client_negotiation.cc:769] Sending connection context
0807 23:01:34.867370 (+ 197us) client_negotiation.cc:240] Negotiation successful
0807 23:01:34.867783 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":286,"threads_started":1}
I20260807 23:01:34.870519 25092 heartbeater.cc:347] Connected to a master server at 127.25.94.62:36359
I20260807 23:01:34.871055 25092 heartbeater.cc:464] Registering TS with master...
I20260807 23:01:34.871868 25092 heartbeater.cc:511] Master 127.25.94.62:36359 requested a full tablet report, sending...
I20260807 23:01:34.874059 24789 ts_manager.cc:194] Registered new tserver with Master: 91c7e3c5fe8d4c9c90622ec014280afd (127.25.94.1:44931)
I20260807 23:01:34.875527 24789 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.94.1:55255
W20260807 23:01:35.214551 25096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 23:01:35.214882 25096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 23:01:35.215025 25096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 23:01:35.222792 25096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 23:01:35.222908 25096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 23:01:35.223088 25096 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.2
I20260807 23:01:35.232923 25096 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:33073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786143361779964-25976-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786143361779964-25976-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786143361779964-25976-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.94.2
--webserver_port=0
--tserver_master_addrs=127.25.94.62:36359
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.2
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786143361779964-25976-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 23:01:35.234321 25096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 23:01:35.236356 25096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 23:01:35.250320 25108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 23:01:35.250389 25111 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 23:01:35.250414 25109 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 23:01:35.251613 25096 server_base.cc:1034] running on GCE node
I20260807 23:01:35.252365 25096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 23:01:35.253471 25096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 23:01:35.254686 25096 hybrid_clock.cc:648] HybridClock initialized: now 1786143695254614 us; error 57 us; skew 500 ppm
I20260807 23:01:35.255053 25096 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 23:01:35.257474 25096 webserver.cc:460] Webserver started at http://127.25.94.2:42391/ using document root <none> and password file <none>
I20260807 23:01:35.258184 25096 fs_manager.cc:359] Metadata directory not provided
I20260807 23:01:35.258309 25096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 23:01:35.258642 25096 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 23:01:35.261653 25096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786143361779964-25976-0/minicluster-data/ts-1/data/instance:
uuid: "18db06903fbf446b87c3dc4373fb63d4"
format_stamp: "Formatted at 2026-08-07 23:01:35 on dist-test-slave-z8x9"
server_key: "dc4e2c90633574a0c4e45693c16c4f72"
server_key_iv: "147b3e089cb7f88d6855a22675d8669c"
server_key_version: "encryptionkey@0"
I20260807 23:01:35.262461 25096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786143361779964-25976-0/minicluster-data/ts-1/wal/instance:
uuid: "18db06903fbf446b87c3dc4373fb63d4"
format_stamp: "Formatted at 2026-08-07 23:01:35 on dist-test-slave-z8x9"
server_key: "dc4e2c90633574a0c4e45693c16c4f72"
server_key_iv: "147b3e089cb7f88d6855a22675d8669c"
server_key_version: "encryptionkey@0"
I20260807 23:01:35.267941 25096 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260807 23:01:35.271878 25120 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 23:01:35.273531 25096 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260807 23:01:35.273732 25096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786143361779964-25976-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786143361779964-25976-0/minicluster-data/ts-1/wal
uuid: "18db06903fbf446b87c3dc4373fb63d4"
format_stamp: "Formatted at 2026-08-07 23:01:35 on dist-test-slave-z8x9"
server_key: "dc4e2c90633574a0c4e45693c16c4f72"
server_key_iv: "147b3e089cb7f88d6855a22675d8669c"
server_key_version: "encryptionkey@0"
I20260807 23:01:35.273939 25096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786143361779964-25976-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786143361779964-25976-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786143361779964-25976-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 23:01:35.299283 25096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 23:01:35.300287 25096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 23:01:35.300598 25096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 23:01:35.301793 25096 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 23:01:35.303437 25096 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 23:01:35.303555 25096 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 23:01:35.303699 25096 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 23:01:35.303802 25096 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 23:01:35.339677 25096 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.2:34177
I20260807 23:01:35.339773 25298 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.2:34177 every 8 connection(s)
I20260807 23:01:35.340996 25096 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786143361779964-25976-0/minicluster-data/ts-1/data/info.pb
I20260807 23:01:35.349563 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 25096
I20260807 23:01:35.349779 25976 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786143361779964-25976-0/minicluster-data/ts-1/wal/instance
I20260807 23:01:35.350286 25976 external_mini_cluster.cc:1428] Setting key f66406ba491f5e8aeece7cb9eb466558
I20260807 23:01:35.353591 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786143361779964-25976-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786143361779964-25976-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786143361779964-25976-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786143361779964-25976-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.3:0
--local_ip_for_outbound_sockets=127.25.94.3
--webserver_interface=127.25.94.3
--webserver_port=0
--tserver_master_addrs=127.25.94.62:36359
--builtin_ntp_servers=127.25.94.20:33073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260807 23:01:35.354645 25094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:35.343651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.2:36729 (local address 127.25.94.62:36359)
0807 23:01:35.343883 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0807 23:01:35.343890 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 23:01:35.345291 (+ 1401us) server_negotiation.cc:407] Connection header received
0807 23:01:35.347112 (+ 1821us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:01:35.347117 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:01:35.347192 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:01:35.347291 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:01:35.348372 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:35.349216 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:35.351373 (+ 2157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:35.351836 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:35.351934 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:35.353671 (+ 1737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:01:35.353679 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:01:35.353684 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 23:01:35.353707 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:01:35.353765 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 23:01:35.353788 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:01:35.353884 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0807 23:01:35.354124 (+ 240us) server_negotiation.cc:299] Negotiation successful
0807 23:01:35.354239 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260807 23:01:35.355590 25303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:35.344158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:36359 (local address 127.25.94.2:36729)
0807 23:01:35.345072 (+ 914us) negotiation.cc:107] Waiting for socket to connect
0807 23:01:35.345118 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0807 23:01:35.346822 (+ 1704us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 23:01:35.347489 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 23:01:35.347497 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 23:01:35.347590 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0807 23:01:35.348186 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:01:35.348203 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:01:35.349383 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:01:35.349387 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:01:35.351218 (+ 1831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:01:35.351232 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:01:35.352077 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:01:35.352086 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:01:35.352451 (+ 365us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:35.353274 (+ 823us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 23:01:35.353277 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0807 23:01:35.353337 (+ 60us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 23:01:35.353443 (+ 106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 23:01:35.353907 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 23:01:35.353916 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 23:01:35.353922 (+ 6us) client_negotiation.cc:769] Sending connection context
0807 23:01:35.354077 (+ 155us) client_negotiation.cc:240] Negotiation successful
0807 23:01:35.354401 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":287,"threads_started":1}
I20260807 23:01:35.357286 25299 heartbeater.cc:347] Connected to a master server at 127.25.94.62:36359
I20260807 23:01:35.357640 25299 heartbeater.cc:464] Registering TS with master...
I20260807 23:01:35.358606 25299 heartbeater.cc:511] Master 127.25.94.62:36359 requested a full tablet report, sending...
I20260807 23:01:35.360349 24789 ts_manager.cc:194] Registered new tserver with Master: 18db06903fbf446b87c3dc4373fb63d4 (127.25.94.2:34177)
I20260807 23:01:35.361212 24789 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.94.2:36729
W20260807 23:01:35.709074 25304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 23:01:35.709455 25304 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 23:01:35.709569 25304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 23:01:35.717127 25304 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 23:01:35.717283 25304 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 23:01:35.717398 25304 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.3
I20260807 23:01:35.727046 25304 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:33073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786143361779964-25976-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786143361779964-25976-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786143361779964-25976-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.94.3
--webserver_port=0
--tserver_master_addrs=127.25.94.62:36359
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.3
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786143361779964-25976-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 23:01:35.728308 25304 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 23:01:35.730140 25304 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 23:01:35.743377 25314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 23:01:35.743474 25315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 23:01:35.743432 25317 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 23:01:35.744798 25304 server_base.cc:1034] running on GCE node
I20260807 23:01:35.745666 25304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 23:01:35.746686 25304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 23:01:35.747929 25304 hybrid_clock.cc:648] HybridClock initialized: now 1786143695747889 us; error 61 us; skew 500 ppm
I20260807 23:01:35.748279 25304 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 23:01:35.750692 25304 webserver.cc:460] Webserver started at http://127.25.94.3:39997/ using document root <none> and password file <none>
I20260807 23:01:35.751425 25304 fs_manager.cc:359] Metadata directory not provided
I20260807 23:01:35.751564 25304 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 23:01:35.751905 25304 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 23:01:35.755451 25304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786143361779964-25976-0/minicluster-data/ts-2/data/instance:
uuid: "9ffda5bba39947daa06817a1358abda8"
format_stamp: "Formatted at 2026-08-07 23:01:35 on dist-test-slave-z8x9"
server_key: "9c33b0860aa964a06eba361430b09956"
server_key_iv: "17aabb976a5b0b2260acaa6f4d7e1f30"
server_key_version: "encryptionkey@0"
I20260807 23:01:35.756583 25304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786143361779964-25976-0/minicluster-data/ts-2/wal/instance:
uuid: "9ffda5bba39947daa06817a1358abda8"
format_stamp: "Formatted at 2026-08-07 23:01:35 on dist-test-slave-z8x9"
server_key: "9c33b0860aa964a06eba361430b09956"
server_key_iv: "17aabb976a5b0b2260acaa6f4d7e1f30"
server_key_version: "encryptionkey@0"
I20260807 23:01:35.762318 25304 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260807 23:01:35.766121 25328 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 23:01:35.767498 25304 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 23:01:35.767660 25304 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786143361779964-25976-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786143361779964-25976-0/minicluster-data/ts-2/wal
uuid: "9ffda5bba39947daa06817a1358abda8"
format_stamp: "Formatted at 2026-08-07 23:01:35 on dist-test-slave-z8x9"
server_key: "9c33b0860aa964a06eba361430b09956"
server_key_iv: "17aabb976a5b0b2260acaa6f4d7e1f30"
server_key_version: "encryptionkey@0"
I20260807 23:01:35.767891 25304 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786143361779964-25976-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786143361779964-25976-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786143361779964-25976-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 23:01:35.783423 25304 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 23:01:35.784264 25304 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 23:01:35.784548 25304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 23:01:35.785636 25304 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 23:01:35.787178 25304 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 23:01:35.787308 25304 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 23:01:35.787456 25304 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 23:01:35.787549 25304 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 23:01:35.827056 25304 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.3:38109
I20260807 23:01:35.827111 25518 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.3:38109 every 8 connection(s)
I20260807 23:01:35.828503 25304 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786143361779964-25976-0/minicluster-data/ts-2/data/info.pb
I20260807 23:01:35.837466 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 25304
I20260807 23:01:35.837670 25976 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786143361779964-25976-0/minicluster-data/ts-2/wal/instance
I20260807 23:01:35.838107 25976 external_mini_cluster.cc:1428] Setting key b6199aac20834e8a44901c3e1a9ab37c
I20260807 23:01:35.841398 25094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:35.831078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.3:46123 (local address 127.25.94.62:36359)
0807 23:01:35.831252 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0807 23:01:35.831256 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 23:01:35.832570 (+ 1314us) server_negotiation.cc:407] Connection header received
0807 23:01:35.834469 (+ 1899us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:01:35.834475 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:01:35.834569 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:01:35.834677 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:01:35.835733 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:35.836469 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:35.838195 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:35.838684 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:35.838823 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:35.840480 (+ 1657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:01:35.840487 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:01:35.840492 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 23:01:35.840517 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:01:35.840550 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 23:01:35.840562 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:01:35.840655 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0807 23:01:35.840880 (+ 225us) server_negotiation.cc:299] Negotiation successful
0807 23:01:35.840977 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260807 23:01:35.842332 25521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:35.831545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:36359 (local address 127.25.94.3:46123)
0807 23:01:35.832418 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0807 23:01:35.832460 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0807 23:01:35.834186 (+ 1726us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 23:01:35.834924 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 23:01:35.834932 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 23:01:35.835016 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0807 23:01:35.835565 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:01:35.835582 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:01:35.836644 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:01:35.836648 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:01:35.838042 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:01:35.838055 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:01:35.838943 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:01:35.838951 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:01:35.839298 (+ 347us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:35.840165 (+ 867us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 23:01:35.840168 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0807 23:01:35.840207 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 23:01:35.840280 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 23:01:35.840657 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 23:01:35.840671 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 23:01:35.840680 (+ 9us) client_negotiation.cc:769] Sending connection context
0807 23:01:35.840869 (+ 189us) client_negotiation.cc:240] Negotiation successful
0807 23:01:35.841267 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":289,"threads_started":1}
I20260807 23:01:35.843751 25519 heartbeater.cc:347] Connected to a master server at 127.25.94.62:36359
I20260807 23:01:35.844015 25519 heartbeater.cc:464] Registering TS with master...
I20260807 23:01:35.844704 25519 heartbeater.cc:511] Master 127.25.94.62:36359 requested a full tablet report, sending...
I20260807 23:01:35.846063 24789 ts_manager.cc:194] Registered new tserver with Master: 9ffda5bba39947daa06817a1358abda8 (127.25.94.3:38109)
I20260807 23:01:35.846889 24789 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.94.3:46123
I20260807 23:01:35.853340 25976 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 23:01:35.863879 25094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:35.857587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42320 (local address 127.25.94.62:36359)
0807 23:01:35.857770 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0807 23:01:35.857778 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 23:01:35.858221 (+ 443us) server_negotiation.cc:407] Connection header received
0807 23:01:35.858422 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:01:35.858429 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:01:35.858496 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:01:35.858600 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:01:35.859095 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:35.859785 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:35.861316 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:35.861749 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:35.861887 (+ 138us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:35.863074 (+ 1187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:01:35.863079 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:01:35.863082 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 23:01:35.863100 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:01:35.863131 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 23:01:35.863144 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:01:35.863313 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0807 23:01:35.863511 (+ 198us) server_negotiation.cc:299] Negotiation successful
0807 23:01:35.863568 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260807 23:01:35.867856 24789 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42320:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 23:01:35.875617 24789 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 23:01:35.878931 25092 heartbeater.cc:503] Master 127.25.94.62:36359 was elected leader, sending a full tablet report...
I20260807 23:01:35.899353 25549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:35.888306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41714 (local address 127.25.94.2:34177)
0807 23:01:35.888912 (+ 606us) server_negotiation.cc:206] Beginning negotiation
0807 23:01:35.888923 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 23:01:35.889679 (+ 756us) server_negotiation.cc:407] Connection header received
0807 23:01:35.890094 (+ 415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:01:35.890103 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:01:35.890372 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:01:35.890549 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 23:01:35.891670 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:35.893282 (+ 1612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:35.896097 (+ 2815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:35.897682 (+ 1585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:35.897792 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:35.898039 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0807 23:01:35.898740 (+ 701us) server_negotiation.cc:299] Negotiation successful
0807 23:01:35.898959 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":431,"thread_start_us":300,"threads_started":1}
I20260807 23:01:35.900177 25547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:35.888051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.2:34177 (local address 127.0.0.1:41714)
0807 23:01:35.889484 (+ 1433us) negotiation.cc:107] Waiting for socket to connect
0807 23:01:35.889593 (+ 109us) client_negotiation.cc:174] Beginning negotiation
0807 23:01:35.889780 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 23:01:35.890929 (+ 1149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 23:01:35.890936 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 23:01:35.890968 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 23:01:35.891470 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:01:35.891487 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:01:35.893471 (+ 1984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:01:35.893476 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:01:35.895861 (+ 2385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:01:35.895875 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:01:35.898243 (+ 2368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:01:35.898251 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:01:35.898476 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:35.898527 (+ 51us) client_negotiation.cc:769] Sending connection context
0807 23:01:35.899630 (+ 1103us) client_negotiation.cc:240] Negotiation successful
0807 23:01:35.899865 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1145,"thread_start_us":1816,"threads_started":1}
I20260807 23:01:35.902369 25550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:35.888770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.3:38109 (local address 127.0.0.1:33212)
0807 23:01:35.889484 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0807 23:01:35.889564 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0807 23:01:35.889778 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 23:01:35.892357 (+ 2579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 23:01:35.892362 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 23:01:35.892384 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 23:01:35.892763 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:01:35.892776 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:01:35.896145 (+ 3369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:01:35.896152 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:01:35.898315 (+ 2163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:01:35.898328 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:01:35.901570 (+ 3242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:01:35.901577 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:01:35.901766 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:35.901813 (+ 47us) client_negotiation.cc:769] Sending connection context
0807 23:01:35.901911 (+ 98us) client_negotiation.cc:240] Negotiation successful
0807 23:01:35.902045 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":399,"threads_started":1}
I20260807 23:01:35.902596 25551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:35.889137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33212 (local address 127.25.94.3:38109)
0807 23:01:35.891057 (+ 1920us) server_negotiation.cc:206] Beginning negotiation
0807 23:01:35.891069 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 23:01:35.891104 (+ 35us) server_negotiation.cc:407] Connection header received
0807 23:01:35.891197 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:01:35.891206 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:01:35.891442 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:01:35.892971 (+ 1529us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 23:01:35.893256 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:35.895999 (+ 2743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:35.899793 (+ 3794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:35.901362 (+ 1569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:35.901506 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:35.901777 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0807 23:01:35.902018 (+ 241us) server_negotiation.cc:299] Negotiation successful
0807 23:01:35.902191 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1735,"thread_start_us":1706,"threads_started":1}
I20260807 23:01:35.905141 25197 tablet_service.cc:1467] Processing CreateTablet for tablet 753e692830cf4d78a9dc2cd3b205304f (DEFAULT_TABLE table=test-table [id=08255138b94540778aca6412f7666af7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 23:01:35.906612 25197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 753e692830cf4d78a9dc2cd3b205304f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 23:01:35.906265 25411 tablet_service.cc:1467] Processing CreateTablet for tablet 753e692830cf4d78a9dc2cd3b205304f (DEFAULT_TABLE table=test-table [id=08255138b94540778aca6412f7666af7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 23:01:35.907720 25411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 753e692830cf4d78a9dc2cd3b205304f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 23:01:35.908758 25547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:35.890466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.1:44931 (local address 127.0.0.1:34174)
0807 23:01:35.900408 (+ 9942us) negotiation.cc:107] Waiting for socket to connect
0807 23:01:35.900427 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 23:01:35.900525 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 23:01:35.902000 (+ 1475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 23:01:35.902005 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 23:01:35.902032 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 23:01:35.902185 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:01:35.902195 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:01:35.904380 (+ 2185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:01:35.904385 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:01:35.906149 (+ 1764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:01:35.906166 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:01:35.907984 (+ 1818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:01:35.907991 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:01:35.908178 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:35.908213 (+ 35us) client_negotiation.cc:769] Sending connection context
0807 23:01:35.908324 (+ 111us) client_negotiation.cc:240] Negotiation successful
0807 23:01:35.908497 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9836,"thread_start_us":10262,"threads_started":1}
I20260807 23:01:35.908906 25553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:35.890637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34174 (local address 127.25.94.1:44931)
0807 23:01:35.901364 (+ 10727us) server_negotiation.cc:206] Beginning negotiation
0807 23:01:35.901373 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 23:01:35.901406 (+ 33us) server_negotiation.cc:407] Connection header received
0807 23:01:35.901513 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:01:35.901521 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:01:35.901773 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:01:35.902390 (+ 617us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 23:01:35.902658 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:35.904193 (+ 1535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:35.906339 (+ 2146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:35.907815 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:35.907962 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:35.908237 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0807 23:01:35.908380 (+ 143us) server_negotiation.cc:299] Negotiation successful
0807 23:01:35.908540 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10493,"thread_start_us":10350,"threads_started":1}
I20260807 23:01:35.912840 24974 tablet_service.cc:1467] Processing CreateTablet for tablet 753e692830cf4d78a9dc2cd3b205304f (DEFAULT_TABLE table=test-table [id=08255138b94540778aca6412f7666af7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 23:01:35.914376 24974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 753e692830cf4d78a9dc2cd3b205304f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 23:01:35.918639 25556 tablet_bootstrap.cc:492] T 753e692830cf4d78a9dc2cd3b205304f P 18db06903fbf446b87c3dc4373fb63d4: Bootstrap starting.
I20260807 23:01:35.921993 25556 tablet_bootstrap.cc:654] T 753e692830cf4d78a9dc2cd3b205304f P 18db06903fbf446b87c3dc4373fb63d4: Neither blocks nor log segments found. Creating new log.
I20260807 23:01:35.923393 25556 log.cc:824] T 753e692830cf4d78a9dc2cd3b205304f P 18db06903fbf446b87c3dc4373fb63d4: Log is configured to *not* fsync() on all Append() calls
I20260807 23:01:35.924115 25557 tablet_bootstrap.cc:492] T 753e692830cf4d78a9dc2cd3b205304f P 9ffda5bba39947daa06817a1358abda8: Bootstrap starting.
I20260807 23:01:35.927413 25557 tablet_bootstrap.cc:654] T 753e692830cf4d78a9dc2cd3b205304f P 9ffda5bba39947daa06817a1358abda8: Neither blocks nor log segments found. Creating new log.
I20260807 23:01:35.927412 25556 tablet_bootstrap.cc:492] T 753e692830cf4d78a9dc2cd3b205304f P 18db06903fbf446b87c3dc4373fb63d4: No bootstrap required, opened a new log
I20260807 23:01:35.927731 25556 ts_tablet_manager.cc:1397] T 753e692830cf4d78a9dc2cd3b205304f P 18db06903fbf446b87c3dc4373fb63d4: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260807 23:01:35.928474 25557 log.cc:824] T 753e692830cf4d78a9dc2cd3b205304f P 9ffda5bba39947daa06817a1358abda8: Log is configured to *not* fsync() on all Append() calls
I20260807 23:01:35.929512 25560 tablet_bootstrap.cc:492] T 753e692830cf4d78a9dc2cd3b205304f P 91c7e3c5fe8d4c9c90622ec014280afd: Bootstrap starting.
I20260807 23:01:35.933295 25560 tablet_bootstrap.cc:654] T 753e692830cf4d78a9dc2cd3b205304f P 91c7e3c5fe8d4c9c90622ec014280afd: Neither blocks nor log segments found. Creating new log.
I20260807 23:01:35.933816 25557 tablet_bootstrap.cc:492] T 753e692830cf4d78a9dc2cd3b205304f P 9ffda5bba39947daa06817a1358abda8: No bootstrap required, opened a new log
I20260807 23:01:35.934095 25557 ts_tablet_manager.cc:1397] T 753e692830cf4d78a9dc2cd3b205304f P 9ffda5bba39947daa06817a1358abda8: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260807 23:01:35.934540 25560 log.cc:824] T 753e692830cf4d78a9dc2cd3b205304f P 91c7e3c5fe8d4c9c90622ec014280afd: Log is configured to *not* fsync() on all Append() calls
I20260807 23:01:35.937364 25556 raft_consensus.cc:348] T 753e692830cf4d78a9dc2cd3b205304f P 18db06903fbf446b87c3dc4373fb63d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18db06903fbf446b87c3dc4373fb63d4" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 34177 } } peers { permanent_uuid: "91c7e3c5fe8d4c9c90622ec014280afd" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44931 } } peers { permanent_uuid: "9ffda5bba39947daa06817a1358abda8" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 38109 } }
I20260807 23:01:35.937721 25560 tablet_bootstrap.cc:492] T 753e692830cf4d78a9dc2cd3b205304f P 91c7e3c5fe8d4c9c90622ec014280afd: No bootstrap required, opened a new log
I20260807 23:01:35.937800 25556 raft_consensus.cc:374] T 753e692830cf4d78a9dc2cd3b205304f P 18db06903fbf446b87c3dc4373fb63d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 23:01:35.937913 25556 raft_consensus.cc:729] T 753e692830cf4d78a9dc2cd3b205304f P 18db06903fbf446b87c3dc4373fb63d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18db06903fbf446b87c3dc4373fb63d4, State: Initialized, Role: FOLLOWER
I20260807 23:01:35.937973 25560 ts_tablet_manager.cc:1397] T 753e692830cf4d78a9dc2cd3b205304f P 91c7e3c5fe8d4c9c90622ec014280afd: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20260807 23:01:35.938539 25556 consensus_queue.cc:260] T 753e692830cf4d78a9dc2cd3b205304f P 18db06903fbf446b87c3dc4373fb63d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18db06903fbf446b87c3dc4373fb63d4" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 34177 } } peers { permanent_uuid: "91c7e3c5fe8d4c9c90622ec014280afd" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44931 } } peers { permanent_uuid: "9ffda5bba39947daa06817a1358abda8" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 38109 } }
I20260807 23:01:35.939627 25299 heartbeater.cc:503] Master 127.25.94.62:36359 was elected leader, sending a full tablet report...
I20260807 23:01:35.940297 25556 ts_tablet_manager.cc:1428] T 753e692830cf4d78a9dc2cd3b205304f P 18db06903fbf446b87c3dc4373fb63d4: Time spent starting tablet: real 0.012s user 0.009s sys 0.005s
I20260807 23:01:35.946022 25557 raft_consensus.cc:348] T 753e692830cf4d78a9dc2cd3b205304f P 9ffda5bba39947daa06817a1358abda8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18db06903fbf446b87c3dc4373fb63d4" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 34177 } } peers { permanent_uuid: "91c7e3c5fe8d4c9c90622ec014280afd" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44931 } } peers { permanent_uuid: "9ffda5bba39947daa06817a1358abda8" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 38109 } }
I20260807 23:01:35.946497 25557 raft_consensus.cc:374] T 753e692830cf4d78a9dc2cd3b205304f P 9ffda5bba39947daa06817a1358abda8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 23:01:35.946610 25557 raft_consensus.cc:729] T 753e692830cf4d78a9dc2cd3b205304f P 9ffda5bba39947daa06817a1358abda8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ffda5bba39947daa06817a1358abda8, State: Initialized, Role: FOLLOWER
I20260807 23:01:35.947233 25557 consensus_queue.cc:260] T 753e692830cf4d78a9dc2cd3b205304f P 9ffda5bba39947daa06817a1358abda8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18db06903fbf446b87c3dc4373fb63d4" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 34177 } } peers { permanent_uuid: "91c7e3c5fe8d4c9c90622ec014280afd" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44931 } } peers { permanent_uuid: "9ffda5bba39947daa06817a1358abda8" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 38109 } }
I20260807 23:01:35.948279 25519 heartbeater.cc:503] Master 127.25.94.62:36359 was elected leader, sending a full tablet report...
I20260807 23:01:35.948280 25560 raft_consensus.cc:348] T 753e692830cf4d78a9dc2cd3b205304f P 91c7e3c5fe8d4c9c90622ec014280afd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18db06903fbf446b87c3dc4373fb63d4" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 34177 } } peers { permanent_uuid: "91c7e3c5fe8d4c9c90622ec014280afd" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44931 } } peers { permanent_uuid: "9ffda5bba39947daa06817a1358abda8" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 38109 } }
I20260807 23:01:35.948709 25560 raft_consensus.cc:374] T 753e692830cf4d78a9dc2cd3b205304f P 91c7e3c5fe8d4c9c90622ec014280afd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 23:01:35.948827 25560 raft_consensus.cc:729] T 753e692830cf4d78a9dc2cd3b205304f P 91c7e3c5fe8d4c9c90622ec014280afd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91c7e3c5fe8d4c9c90622ec014280afd, State: Initialized, Role: FOLLOWER
I20260807 23:01:35.949065 25557 ts_tablet_manager.cc:1428] T 753e692830cf4d78a9dc2cd3b205304f P 9ffda5bba39947daa06817a1358abda8: Time spent starting tablet: real 0.015s user 0.010s sys 0.004s
I20260807 23:01:35.949416 25560 consensus_queue.cc:260] T 753e692830cf4d78a9dc2cd3b205304f P 91c7e3c5fe8d4c9c90622ec014280afd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18db06903fbf446b87c3dc4373fb63d4" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 34177 } } peers { permanent_uuid: "91c7e3c5fe8d4c9c90622ec014280afd" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44931 } } peers { permanent_uuid: "9ffda5bba39947daa06817a1358abda8" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 38109 } }
I20260807 23:01:35.950866 25560 ts_tablet_manager.cc:1428] T 753e692830cf4d78a9dc2cd3b205304f P 91c7e3c5fe8d4c9c90622ec014280afd: Time spent starting tablet: real 0.013s user 0.011s sys 0.002s
I20260807 23:01:36.030128 25565 raft_consensus.cc:482] T 753e692830cf4d78a9dc2cd3b205304f P 18db06903fbf446b87c3dc4373fb63d4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 23:01:36.030503 25565 raft_consensus.cc:504] T 753e692830cf4d78a9dc2cd3b205304f P 18db06903fbf446b87c3dc4373fb63d4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18db06903fbf446b87c3dc4373fb63d4" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 34177 } } peers { permanent_uuid: "91c7e3c5fe8d4c9c90622ec014280afd" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44931 } } peers { permanent_uuid: "9ffda5bba39947daa06817a1358abda8" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 38109 } }
I20260807 23:01:36.032305 25565 leader_election.cc:288] T 753e692830cf4d78a9dc2cd3b205304f P 18db06903fbf446b87c3dc4373fb63d4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 91c7e3c5fe8d4c9c90622ec014280afd (127.25.94.1:44931), 9ffda5bba39947daa06817a1358abda8 (127.25.94.3:38109)
I20260807 23:01:36.039376 25553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:36.032880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.2:39427 (local address 127.25.94.1:44931)
0807 23:01:36.033196 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0807 23:01:36.033203 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 23:01:36.033808 (+ 605us) server_negotiation.cc:407] Connection header received
0807 23:01:36.034028 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:01:36.034032 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:01:36.034103 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:01:36.034208 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 23:01:36.034842 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:36.035572 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:36.037577 (+ 2005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:36.038369 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:36.038515 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:36.038627 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0807 23:01:36.038947 (+ 320us) server_negotiation.cc:299] Negotiation successful
0807 23:01:36.039089 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260807 23:01:36.039458 25571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:36.032515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.1:44931 (local address 127.25.94.2:39427)
0807 23:01:36.033633 (+ 1118us) negotiation.cc:107] Waiting for socket to connect
0807 23:01:36.033655 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 23:01:36.033847 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 23:01:36.034227 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 23:01:36.034231 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 23:01:36.034256 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 23:01:36.034675 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:01:36.034686 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:01:36.035715 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:01:36.035728 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:01:36.037363 (+ 1635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:01:36.037377 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:01:36.038576 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:01:36.038582 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:01:36.038778 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:36.038803 (+ 25us) client_negotiation.cc:769] Sending connection context
0807 23:01:36.038916 (+ 113us) client_negotiation.cc:240] Negotiation successful
0807 23:01:36.039045 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":916,"mutex_wait_us":29,"thread_start_us":646,"threads_started":1}
I20260807 23:01:36.039469 25551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:36.032882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.2:49027 (local address 127.25.94.3:38109)
0807 23:01:36.033361 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0807 23:01:36.033369 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 23:01:36.033914 (+ 545us) server_negotiation.cc:407] Connection header received
0807 23:01:36.034125 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:01:36.034129 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:01:36.034189 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:01:36.034289 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 23:01:36.034985 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:36.035707 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:36.037700 (+ 1993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:36.038496 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:36.038656 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:36.038761 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0807 23:01:36.039054 (+ 293us) server_negotiation.cc:299] Negotiation successful
0807 23:01:36.039243 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343}
I20260807 23:01:36.039532 25572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:36.032768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.3:38109 (local address 127.25.94.2:49027)
0807 23:01:36.033795 (+ 1027us) negotiation.cc:107] Waiting for socket to connect
0807 23:01:36.033813 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 23:01:36.033982 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 23:01:36.034319 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 23:01:36.034324 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 23:01:36.034363 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 23:01:36.034831 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:01:36.034840 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:01:36.035863 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:01:36.035868 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:01:36.037498 (+ 1630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:01:36.037511 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:01:36.038717 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:01:36.038727 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:01:36.038886 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:36.038908 (+ 22us) client_negotiation.cc:769] Sending connection context
0807 23:01:36.039010 (+ 102us) client_negotiation.cc:240] Negotiation successful
0807 23:01:36.039144 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":887,"spinlock_wait_cycles":17792,"thread_start_us":705,"threads_started":1}
I20260807 23:01:36.040580 25007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "753e692830cf4d78a9dc2cd3b205304f" candidate_uuid: "18db06903fbf446b87c3dc4373fb63d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91c7e3c5fe8d4c9c90622ec014280afd" is_pre_election: true
I20260807 23:01:36.040791 25446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "753e692830cf4d78a9dc2cd3b205304f" candidate_uuid: "18db06903fbf446b87c3dc4373fb63d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ffda5bba39947daa06817a1358abda8" is_pre_election: true
I20260807 23:01:36.041302 25007 raft_consensus.cc:2454] T 753e692830cf4d78a9dc2cd3b205304f P 91c7e3c5fe8d4c9c90622ec014280afd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18db06903fbf446b87c3dc4373fb63d4 in term 0.
I20260807 23:01:36.041325 25446 raft_consensus.cc:2454] T 753e692830cf4d78a9dc2cd3b205304f P 9ffda5bba39947daa06817a1358abda8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18db06903fbf446b87c3dc4373fb63d4 in term 0.
I20260807 23:01:36.042119 25124 leader_election.cc:302] T 753e692830cf4d78a9dc2cd3b205304f P 18db06903fbf446b87c3dc4373fb63d4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 18db06903fbf446b87c3dc4373fb63d4, 91c7e3c5fe8d4c9c90622ec014280afd, 9ffda5bba39947daa06817a1358abda8; no voters:
I20260807 23:01:36.042634 25565 raft_consensus.cc:2781] T 753e692830cf4d78a9dc2cd3b205304f P 18db06903fbf446b87c3dc4373fb63d4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 23:01:36.042837 25565 raft_consensus.cc:482] T 753e692830cf4d78a9dc2cd3b205304f P 18db06903fbf446b87c3dc4373fb63d4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 23:01:36.042963 25565 raft_consensus.cc:3037] T 753e692830cf4d78a9dc2cd3b205304f P 18db06903fbf446b87c3dc4373fb63d4 [term 0 FOLLOWER]: Advancing to term 1
I20260807 23:01:36.045305 25565 raft_consensus.cc:504] T 753e692830cf4d78a9dc2cd3b205304f P 18db06903fbf446b87c3dc4373fb63d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18db06903fbf446b87c3dc4373fb63d4" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 34177 } } peers { permanent_uuid: "91c7e3c5fe8d4c9c90622ec014280afd" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44931 } } peers { permanent_uuid: "9ffda5bba39947daa06817a1358abda8" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 38109 } }
I20260807 23:01:36.046080 25565 leader_election.cc:288] T 753e692830cf4d78a9dc2cd3b205304f P 18db06903fbf446b87c3dc4373fb63d4 [CANDIDATE]: Term 1 election: Requested vote from peers 91c7e3c5fe8d4c9c90622ec014280afd (127.25.94.1:44931), 9ffda5bba39947daa06817a1358abda8 (127.25.94.3:38109)
I20260807 23:01:36.046612 25007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "753e692830cf4d78a9dc2cd3b205304f" candidate_uuid: "18db06903fbf446b87c3dc4373fb63d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91c7e3c5fe8d4c9c90622ec014280afd"
I20260807 23:01:36.046677 25446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "753e692830cf4d78a9dc2cd3b205304f" candidate_uuid: "18db06903fbf446b87c3dc4373fb63d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ffda5bba39947daa06817a1358abda8"
I20260807 23:01:36.046845 25007 raft_consensus.cc:3037] T 753e692830cf4d78a9dc2cd3b205304f P 91c7e3c5fe8d4c9c90622ec014280afd [term 0 FOLLOWER]: Advancing to term 1
I20260807 23:01:36.046885 25446 raft_consensus.cc:3037] T 753e692830cf4d78a9dc2cd3b205304f P 9ffda5bba39947daa06817a1358abda8 [term 0 FOLLOWER]: Advancing to term 1
I20260807 23:01:36.050263 25007 raft_consensus.cc:2454] T 753e692830cf4d78a9dc2cd3b205304f P 91c7e3c5fe8d4c9c90622ec014280afd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18db06903fbf446b87c3dc4373fb63d4 in term 1.
I20260807 23:01:36.050256 25446 raft_consensus.cc:2454] T 753e692830cf4d78a9dc2cd3b205304f P 9ffda5bba39947daa06817a1358abda8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18db06903fbf446b87c3dc4373fb63d4 in term 1.
I20260807 23:01:36.050851 25122 leader_election.cc:302] T 753e692830cf4d78a9dc2cd3b205304f P 18db06903fbf446b87c3dc4373fb63d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18db06903fbf446b87c3dc4373fb63d4, 91c7e3c5fe8d4c9c90622ec014280afd; no voters:
I20260807 23:01:36.051283 25565 raft_consensus.cc:2781] T 753e692830cf4d78a9dc2cd3b205304f P 18db06903fbf446b87c3dc4373fb63d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 23:01:36.051803 25565 raft_consensus.cc:686] T 753e692830cf4d78a9dc2cd3b205304f P 18db06903fbf446b87c3dc4373fb63d4 [term 1 LEADER]: Becoming Leader. State: Replica: 18db06903fbf446b87c3dc4373fb63d4, State: Running, Role: LEADER
I20260807 23:01:36.052296 25565 consensus_queue.cc:237] T 753e692830cf4d78a9dc2cd3b205304f P 18db06903fbf446b87c3dc4373fb63d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18db06903fbf446b87c3dc4373fb63d4" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 34177 } } peers { permanent_uuid: "91c7e3c5fe8d4c9c90622ec014280afd" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44931 } } peers { permanent_uuid: "9ffda5bba39947daa06817a1358abda8" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 38109 } }
I20260807 23:01:36.057125 24788 catalog_manager.cc:5495] T 753e692830cf4d78a9dc2cd3b205304f P 18db06903fbf446b87c3dc4373fb63d4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 18db06903fbf446b87c3dc4373fb63d4 (127.25.94.2). New cstate: current_term: 1 leader_uuid: "18db06903fbf446b87c3dc4373fb63d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18db06903fbf446b87c3dc4373fb63d4" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 34177 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91c7e3c5fe8d4c9c90622ec014280afd" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ffda5bba39947daa06817a1358abda8" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 38109 } health_report { overall_health: UNKNOWN } } }
W20260807 23:01:36.080399 25520 tablet.cc:2369] T 753e692830cf4d78a9dc2cd3b205304f P 9ffda5bba39947daa06817a1358abda8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 23:01:36.093389 25301 tablet.cc:2369] T 753e692830cf4d78a9dc2cd3b205304f P 18db06903fbf446b87c3dc4373fb63d4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 23:01:36.099602 25549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:36.093395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41724 (local address 127.25.94.2:34177)
0807 23:01:36.093627 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0807 23:01:36.093636 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 23:01:36.093660 (+ 24us) server_negotiation.cc:407] Connection header received
0807 23:01:36.093758 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:01:36.093765 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:01:36.093908 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:01:36.094072 (+ 164us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 23:01:36.094565 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:36.095360 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:36.096899 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:36.097484 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:36.097599 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:36.098189 (+ 590us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 23:01:36.098720 (+ 531us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 23:01:36.098874 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0807 23:01:36.099139 (+ 265us) server_negotiation.cc:299] Negotiation successful
0807 23:01:36.099336 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260807 23:01:36.107344 25093 tablet.cc:2369] T 753e692830cf4d78a9dc2cd3b205304f P 91c7e3c5fe8d4c9c90622ec014280afd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 23:01:36.109615 25007 raft_consensus.cc:1261] T 753e692830cf4d78a9dc2cd3b205304f P 91c7e3c5fe8d4c9c90622ec014280afd [term 1 FOLLOWER]: Refusing update from remote peer 18db06903fbf446b87c3dc4373fb63d4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 23:01:36.109613 25446 raft_consensus.cc:1261] T 753e692830cf4d78a9dc2cd3b205304f P 9ffda5bba39947daa06817a1358abda8 [term 1 FOLLOWER]: Refusing update from remote peer 18db06903fbf446b87c3dc4373fb63d4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 23:01:36.110765 25565 consensus_queue.cc:1035] T 753e692830cf4d78a9dc2cd3b205304f P 18db06903fbf446b87c3dc4373fb63d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ffda5bba39947daa06817a1358abda8" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 38109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 23:01:36.111197 25574 consensus_queue.cc:1035] T 753e692830cf4d78a9dc2cd3b205304f P 18db06903fbf446b87c3dc4373fb63d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91c7e3c5fe8d4c9c90622ec014280afd" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 44931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 23:01:36.122676 25578 mvcc.cc:205] Tried to move back new op lower bound from 7316044579252396032 to 7316044579034169344. Current Snapshot: MvccSnapshot[applied={T|T < 7316044579252396032}]
I20260807 23:01:36.127313 25580 mvcc.cc:205] Tried to move back new op lower bound from 7316044579252396032 to 7316044579034169344. Current Snapshot: MvccSnapshot[applied={T|T < 7316044579252396032}]
I20260807 23:01:36.145790 24788 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42320:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 23:01:36.146153 24788 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42320:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 23:01:36.150446 24788 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a3fd7634127a4553937a163952df0d7b: Sending DeleteTablet for 3 replicas of tablet 753e692830cf4d78a9dc2cd3b205304f
I20260807 23:01:36.151886 25197 tablet_service.cc:1514] Processing DeleteTablet for tablet 753e692830cf4d78a9dc2cd3b205304f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 23:01:36 UTC) from {username='slave'} at 127.0.0.1:41714
I20260807 23:01:36.151952 24974 tablet_service.cc:1514] Processing DeleteTablet for tablet 753e692830cf4d78a9dc2cd3b205304f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 23:01:36 UTC) from {username='slave'} at 127.0.0.1:34174
I20260807 23:01:36.152002 25411 tablet_service.cc:1514] Processing DeleteTablet for tablet 753e692830cf4d78a9dc2cd3b205304f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 23:01:36 UTC) from {username='slave'} at 127.0.0.1:33212
I20260807 23:01:36.160164 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 24876
I20260807 23:01:36.162075 25601 tablet_replica.cc:330] T 753e692830cf4d78a9dc2cd3b205304f P 9ffda5bba39947daa06817a1358abda8: stopping tablet replica
I20260807 23:01:36.162626 25601 raft_consensus.cc:2229] T 753e692830cf4d78a9dc2cd3b205304f P 9ffda5bba39947daa06817a1358abda8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 23:01:36.163234 25601 raft_consensus.cc:2258] T 753e692830cf4d78a9dc2cd3b205304f P 9ffda5bba39947daa06817a1358abda8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 23:01:36.169335 25600 tablet_replica.cc:330] T 753e692830cf4d78a9dc2cd3b205304f P 18db06903fbf446b87c3dc4373fb63d4: stopping tablet replica
I20260807 23:01:36.170058 25600 raft_consensus.cc:2229] T 753e692830cf4d78a9dc2cd3b205304f P 18db06903fbf446b87c3dc4373fb63d4 [term 1 LEADER]: Raft consensus shutting down.
I20260807 23:01:36.170981 25600 raft_consensus.cc:2258] T 753e692830cf4d78a9dc2cd3b205304f P 18db06903fbf446b87c3dc4373fb63d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 23:01:36.173638 25601 ts_tablet_manager.cc:1905] T 753e692830cf4d78a9dc2cd3b205304f P 9ffda5bba39947daa06817a1358abda8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 23:01:36.174382 25600 ts_tablet_manager.cc:1905] T 753e692830cf4d78a9dc2cd3b205304f P 18db06903fbf446b87c3dc4373fb63d4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 23:01:36.177215 25598 tablet_replica.cc:330] T 753e692830cf4d78a9dc2cd3b205304f P 91c7e3c5fe8d4c9c90622ec014280afd: stopping tablet replica
I20260807 23:01:36.178471 25598 raft_consensus.cc:2229] T 753e692830cf4d78a9dc2cd3b205304f P 91c7e3c5fe8d4c9c90622ec014280afd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 23:01:36.179636 25598 raft_consensus.cc:2258] T 753e692830cf4d78a9dc2cd3b205304f P 91c7e3c5fe8d4c9c90622ec014280afd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 23:01:36.184741 25600 ts_tablet_manager.cc:1918] T 753e692830cf4d78a9dc2cd3b205304f P 18db06903fbf446b87c3dc4373fb63d4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 23:01:36.185117 25600 log.cc:1196] T 753e692830cf4d78a9dc2cd3b205304f P 18db06903fbf446b87c3dc4373fb63d4: Deleting WAL directory at /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786143361779964-25976-0/minicluster-data/ts-1/wal/wals/753e692830cf4d78a9dc2cd3b205304f
I20260807 23:01:36.186085 25600 ts_tablet_manager.cc:1939] T 753e692830cf4d78a9dc2cd3b205304f P 18db06903fbf446b87c3dc4373fb63d4: Deleting consensus metadata
I20260807 23:01:36.187995 25598 ts_tablet_manager.cc:1905] T 753e692830cf4d78a9dc2cd3b205304f P 91c7e3c5fe8d4c9c90622ec014280afd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 23:01:36.188410 24767 catalog_manager.cc:4841] TS 18db06903fbf446b87c3dc4373fb63d4 (127.25.94.2:34177): tablet 753e692830cf4d78a9dc2cd3b205304f (table test-table [id=08255138b94540778aca6412f7666af7]) successfully deleted
I20260807 23:01:36.189808 25601 ts_tablet_manager.cc:1918] T 753e692830cf4d78a9dc2cd3b205304f P 9ffda5bba39947daa06817a1358abda8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 23:01:36.190021 25601 log.cc:1196] T 753e692830cf4d78a9dc2cd3b205304f P 9ffda5bba39947daa06817a1358abda8: Deleting WAL directory at /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786143361779964-25976-0/minicluster-data/ts-2/wal/wals/753e692830cf4d78a9dc2cd3b205304f
I20260807 23:01:36.190589 25601 ts_tablet_manager.cc:1939] T 753e692830cf4d78a9dc2cd3b205304f P 9ffda5bba39947daa06817a1358abda8: Deleting consensus metadata
I20260807 23:01:36.192663 24769 catalog_manager.cc:4841] TS 9ffda5bba39947daa06817a1358abda8 (127.25.94.3:38109): tablet 753e692830cf4d78a9dc2cd3b205304f (table test-table [id=08255138b94540778aca6412f7666af7]) successfully deleted
I20260807 23:01:36.196010 25598 ts_tablet_manager.cc:1918] T 753e692830cf4d78a9dc2cd3b205304f P 91c7e3c5fe8d4c9c90622ec014280afd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 23:01:36.196231 25598 log.cc:1196] T 753e692830cf4d78a9dc2cd3b205304f P 91c7e3c5fe8d4c9c90622ec014280afd: Deleting WAL directory at /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786143361779964-25976-0/minicluster-data/ts-0/wal/wals/753e692830cf4d78a9dc2cd3b205304f
I20260807 23:01:36.197345 25598 ts_tablet_manager.cc:1939] T 753e692830cf4d78a9dc2cd3b205304f P 91c7e3c5fe8d4c9c90622ec014280afd: Deleting consensus metadata
I20260807 23:01:36.199481 24767 catalog_manager.cc:4841] TS 91c7e3c5fe8d4c9c90622ec014280afd (127.25.94.1:44931): tablet 753e692830cf4d78a9dc2cd3b205304f (table test-table [id=08255138b94540778aca6412f7666af7]) successfully deleted
I20260807 23:01:36.199636 25553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:36.175794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34178 (local address 127.25.94.1:44931)
0807 23:01:36.176193 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0807 23:01:36.176201 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 23:01:36.189745 (+ 13544us) server_negotiation.cc:407] Connection header received
0807 23:01:36.190370 (+ 625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:01:36.190376 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:01:36.190488 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:01:36.190570 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:01:36.191359 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:36.192466 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:36.194601 (+ 2135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:36.195296 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:36.195396 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:36.197349 (+ 1953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:01:36.197364 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:01:36.197386 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 23:01:36.198519 (+ 1133us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:01:36.198619 (+ 100us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 23:01:36.198673 (+ 54us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:01:36.198773 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0807 23:01:36.199098 (+ 325us) server_negotiation.cc:299] Negotiation successful
0807 23:01:36.199238 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227}
I20260807 23:01:36.200523 25085 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 23:01:36.308964 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 24876
I20260807 23:01:36.344437 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 25096
I20260807 23:01:36.353863 25549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:36.345568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41728 (local address 127.25.94.2:34177)
0807 23:01:36.345800 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0807 23:01:36.345807 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 23:01:36.345836 (+ 29us) server_negotiation.cc:407] Connection header received
0807 23:01:36.346001 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:01:36.346008 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:01:36.346133 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:01:36.346274 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:01:36.346739 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:36.348033 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:36.349938 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:36.350622 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:36.350754 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:36.352026 (+ 1272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:01:36.352038 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:01:36.352061 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 23:01:36.352920 (+ 859us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:01:36.352985 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 23:01:36.353016 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:01:36.353131 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0807 23:01:36.353326 (+ 195us) server_negotiation.cc:299] Negotiation successful
0807 23:01:36.353440 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260807 23:01:36.354779 25293 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 23:01:36.468780 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 25096
I20260807 23:01:36.499699 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 25304
I20260807 23:01:36.506924 25551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:36.500546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33226 (local address 127.25.94.3:38109)
0807 23:01:36.500750 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0807 23:01:36.500755 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 23:01:36.500842 (+ 87us) server_negotiation.cc:407] Connection header received
0807 23:01:36.501067 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:01:36.501072 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:01:36.501161 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:01:36.501353 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:01:36.501747 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:36.502476 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:36.503776 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:36.504203 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:36.504298 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:36.505572 (+ 1274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:01:36.505580 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:01:36.505595 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 23:01:36.506174 (+ 579us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:01:36.506229 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 23:01:36.506250 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:01:36.506339 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0807 23:01:36.506463 (+ 124us) server_negotiation.cc:299] Negotiation successful
0807 23:01:36.506553 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260807 23:01:36.507577 25513 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 23:01:36.597267 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 25304
I20260807 23:01:36.623612 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 24742
I20260807 23:01:36.624837 24829 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 23:01:36.741403 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 24742
2026-08-07T23:01:36Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (3026 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786143361779964-25976-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 23:01:36 dist-test-slave-z8x9 krb5kdc[25610](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 23:01:36 dist-test-slave-z8x9 krb5kdc[25610](info): set up 2 sockets
Aug 07 23:01:36 dist-test-slave-z8x9 krb5kdc[25610](info): commencing operation
krb5kdc: starting...
W20260807 23:01:38.136301 25976 mini_kdc.cc:121] Time spent starting KDC: real 1.342s user 0.000s sys 0.041s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786143361779964-25976-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786143361779964-25976-0/krb5kdc/test-user.keytab.
Aug 07 23:01:38 dist-test-slave-z8x9 krb5kdc[25610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143698, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T23:01:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T23:01:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.94.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.62.keytab.
Entry for principal kudu/127.25.94.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.62.keytab.
I20260807 23:01:38.484534 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786143361779964-25976-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786143361779964-25976-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786143361779964-25976-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.62:36775
--webserver_interface=127.25.94.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.62.keytab
--principal=kudu/127.25.94.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.20:32943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.94.62:36775
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786143361779964-25976-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786143361779964-25976-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786143361779964-25976-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 23:01:38.842840 25626 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 23:01:38.843130 25626 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 23:01:38.843302 25626 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 23:01:38.850730 25626 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 23:01:38.850863 25626 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 23:01:38.850948 25626 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 23:01:38.851034 25626 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 23:01:38.851125 25626 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 23:01:38.861569 25626 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:32943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786143361779964-25976-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786143361779964-25976-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.94.62:36775
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.62.keytab
--principal=kudu/127.25.94.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.62:36775
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.94.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786143361779964-25976-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 23:01:38.862987 25626 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 23:01:38.864346 25626 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 23:01:38.877614 25639 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 23:01:38.877580 25636 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 23:01:38.877710 25637 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 23:01:38.878827 25626 server_base.cc:1034] running on GCE node
I20260807 23:01:38.879648 25626 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 23:01:38.881130 25626 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 23:01:38.882423 25626 hybrid_clock.cc:648] HybridClock initialized: now 1786143698882381 us; error 30 us; skew 500 ppm
I20260807 23:01:38.882794 25626 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 23:01:38 dist-test-slave-z8x9 krb5kdc[25610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143698, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 23:01:38.889933 25626 init.cc:377] Logged in from keytab as kudu/127.25.94.62@KRBTEST.COM (short username kudu)
I20260807 23:01:38.891537 25626 webserver.cc:460] Webserver started at http://127.25.94.62:35853/ using document root <none> and password file <none>
I20260807 23:01:38.892238 25626 fs_manager.cc:359] Metadata directory not provided
I20260807 23:01:38.892400 25626 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 23:01:38.892715 25626 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 23:01:38.895764 25626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786143361779964-25976-0/minicluster-data/master-0/data/instance:
uuid: "9e82911ca102413b95f33756345c831a"
format_stamp: "Formatted at 2026-08-07 23:01:38 on dist-test-slave-z8x9"
server_key: "158f211819ae7e397e702ba05fac31cb"
server_key_iv: "96e11f7e940f5c48df6a7b643dbcf39e"
server_key_version: "encryptionkey@0"
I20260807 23:01:38.896600 25626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786143361779964-25976-0/minicluster-data/master-0/wal/instance:
uuid: "9e82911ca102413b95f33756345c831a"
format_stamp: "Formatted at 2026-08-07 23:01:38 on dist-test-slave-z8x9"
server_key: "158f211819ae7e397e702ba05fac31cb"
server_key_iv: "96e11f7e940f5c48df6a7b643dbcf39e"
server_key_version: "encryptionkey@0"
I20260807 23:01:38.901901 25626 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260807 23:01:38.905418 25656 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 23:01:38.906683 25626 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260807 23:01:38.906864 25626 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786143361779964-25976-0/minicluster-data/master-0/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786143361779964-25976-0/minicluster-data/master-0/wal
uuid: "9e82911ca102413b95f33756345c831a"
format_stamp: "Formatted at 2026-08-07 23:01:38 on dist-test-slave-z8x9"
server_key: "158f211819ae7e397e702ba05fac31cb"
server_key_iv: "96e11f7e940f5c48df6a7b643dbcf39e"
server_key_version: "encryptionkey@0"
I20260807 23:01:38.907058 25626 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786143361779964-25976-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786143361779964-25976-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786143361779964-25976-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 23:01:38.938301 25626 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 23:01:38.942548 25626 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 23:01:38.942860 25626 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 23:01:38.961768 25626 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.62:36775
I20260807 23:01:38.961798 25734 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.62:36775 every 8 connection(s)
I20260807 23:01:38.963284 25626 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
I20260807 23:01:38.966785 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 25626
I20260807 23:01:38.966984 25976 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786143361779964-25976-0/minicluster-data/master-0/wal/instance
I20260807 23:01:38.967373 25735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 23:01:38.967466 25976 external_mini_cluster.cc:1428] Setting key 3fa50b3233845413545a018a75861be1
I20260807 23:01:38.982357 25735 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e82911ca102413b95f33756345c831a: Bootstrap starting.
Aug 07 23:01:38 dist-test-slave-z8x9 krb5kdc[25610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143698, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.62@KRBTEST.COM
I20260807 23:01:38.986804 25735 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9e82911ca102413b95f33756345c831a: Neither blocks nor log segments found. Creating new log.
I20260807 23:01:38.989292 25735 log.cc:824] T 00000000000000000000000000000000 P 9e82911ca102413b95f33756345c831a: Log is configured to *not* fsync() on all Append() calls
I20260807 23:01:38.993765 25735 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e82911ca102413b95f33756345c831a: No bootstrap required, opened a new log
I20260807 23:01:38.996354 25743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:38.969424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37046 (local address 127.25.94.62:36775)
0807 23:01:38.970277 (+ 853us) server_negotiation.cc:206] Beginning negotiation
0807 23:01:38.970292 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0807 23:01:38.970347 (+ 55us) server_negotiation.cc:407] Connection header received
0807 23:01:38.972510 (+ 2163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:01:38.972540 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:01:38.973077 (+ 537us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:01:38.973582 (+ 505us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:01:38.975198 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:38.977204 (+ 2006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:38.978781 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:38.979940 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:38.980135 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:38.984546 (+ 4411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:01:38.984558 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:01:38.984577 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 23:01:38.984631 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:01:38.990507 (+ 5876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:01:38.992042 (+ 1535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:01:38.992054 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:01:38.992063 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:01:38.992332 (+ 269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:01:38.992779 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:01:38.992787 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:01:38.992791 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:01:38.993579 (+ 788us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:01:38.993921 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0807 23:01:38.994434 (+ 513us) server_negotiation.cc:299] Negotiation successful
0807 23:01:38.994929 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":602,"thread_start_us":343,"threads_started":1}
I20260807 23:01:39.003386 25735 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9e82911ca102413b95f33756345c831a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e82911ca102413b95f33756345c831a" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 36775 } }
I20260807 23:01:39.003793 25735 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9e82911ca102413b95f33756345c831a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 23:01:39.003881 25735 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9e82911ca102413b95f33756345c831a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e82911ca102413b95f33756345c831a, State: Initialized, Role: FOLLOWER
I20260807 23:01:39.004487 25735 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9e82911ca102413b95f33756345c831a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e82911ca102413b95f33756345c831a" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 36775 } }
I20260807 23:01:39.004711 25735 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9e82911ca102413b95f33756345c831a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 23:01:39.004813 25735 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9e82911ca102413b95f33756345c831a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 23:01:39.004941 25735 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9e82911ca102413b95f33756345c831a [term 0 FOLLOWER]: Advancing to term 1
I20260807 23:01:39.007501 25735 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9e82911ca102413b95f33756345c831a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e82911ca102413b95f33756345c831a" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 36775 } }
I20260807 23:01:39.007978 25735 leader_election.cc:302] T 00000000000000000000000000000000 P 9e82911ca102413b95f33756345c831a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9e82911ca102413b95f33756345c831a; no voters:
I20260807 23:01:39.008704 25735 leader_election.cc:288] T 00000000000000000000000000000000 P 9e82911ca102413b95f33756345c831a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 23:01:39.008915 25745 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9e82911ca102413b95f33756345c831a [term 1 FOLLOWER]: Leader election won for term 1
I20260807 23:01:39.009678 25745 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9e82911ca102413b95f33756345c831a [term 1 LEADER]: Becoming Leader. State: Replica: 9e82911ca102413b95f33756345c831a, State: Running, Role: LEADER
I20260807 23:01:39.010349 25745 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9e82911ca102413b95f33756345c831a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e82911ca102413b95f33756345c831a" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 36775 } }
I20260807 23:01:39.011004 25735 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9e82911ca102413b95f33756345c831a [sys.catalog]: configured and running, proceeding with master startup.
I20260807 23:01:39.014117 25748 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e82911ca102413b95f33756345c831a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9e82911ca102413b95f33756345c831a. Latest consensus state: current_term: 1 leader_uuid: "9e82911ca102413b95f33756345c831a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e82911ca102413b95f33756345c831a" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 36775 } } }
I20260807 23:01:39.014129 25746 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e82911ca102413b95f33756345c831a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9e82911ca102413b95f33756345c831a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e82911ca102413b95f33756345c831a" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 36775 } } }
I20260807 23:01:39.014601 25746 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e82911ca102413b95f33756345c831a [sys.catalog]: This master's current role is: LEADER
I20260807 23:01:39.014601 25748 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e82911ca102413b95f33756345c831a [sys.catalog]: This master's current role is: LEADER
I20260807 23:01:39.018043 25758 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 23:01:39.024088 25758 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 23:01:39.033581 25758 catalog_manager.cc:1347] Generated new cluster ID: b6608320141e40cdb0a38c725e7531df
I20260807 23:01:39.033710 25758 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 23:01:39.065685 25758 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 23:01:39.067080 25758 catalog_manager.cc:1504] Loading token signing keys...
I20260807 23:01:39.080962 25758 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9e82911ca102413b95f33756345c831a: Generated new TSK 0
I20260807 23:01:39.081897 25758 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.94.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.1.keytab.
Entry for principal kudu/127.25.94.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.1.keytab.
I20260807 23:01:39.170013 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786143361779964-25976-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786143361779964-25976-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786143361779964-25976-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786143361779964-25976-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.1:0
--local_ip_for_outbound_sockets=127.25.94.1
--webserver_interface=127.25.94.1
--webserver_port=0
--tserver_master_addrs=127.25.94.62:36775
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.1.keytab
--principal=kudu/127.25.94.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.20:32943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786143361779964-25976-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786143361779964-25976-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786143361779964-25976-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 23:01:39.522738 25777 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 23:01:39.523140 25777 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 23:01:39.523306 25777 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 23:01:39.530946 25777 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 23:01:39.531075 25777 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 23:01:39.531261 25777 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.1
I20260807 23:01:39.541030 25777 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:32943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786143361779964-25976-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786143361779964-25976-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.1.keytab
--principal=kudu/127.25.94.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786143361779964-25976-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.94.1
--webserver_port=0
--tserver_master_addrs=127.25.94.62:36775
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.1
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786143361779964-25976-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 23:01:39.542464 25777 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 23:01:39.543920 25777 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 23:01:39.557567 25786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 23:01:39.557663 25784 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 23:01:39.557689 25788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 23:01:39.558791 25777 server_base.cc:1034] running on GCE node
I20260807 23:01:39.559446 25777 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 23:01:39.560499 25777 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 23:01:39.561739 25777 hybrid_clock.cc:648] HybridClock initialized: now 1786143699561699 us; error 58 us; skew 500 ppm
I20260807 23:01:39.562072 25777 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 23:01:39 dist-test-slave-z8x9 krb5kdc[25610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143699, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 23:01:39.565554 25777 init.cc:377] Logged in from keytab as kudu/127.25.94.1@KRBTEST.COM (short username kudu)
I20260807 23:01:39.567196 25777 webserver.cc:460] Webserver started at http://127.25.94.1:37349/ using document root <none> and password file <none>
I20260807 23:01:39.567947 25777 fs_manager.cc:359] Metadata directory not provided
I20260807 23:01:39.568109 25777 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 23:01:39.568459 25777 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 23:01:39.571859 25777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786143361779964-25976-0/minicluster-data/ts-0/data/instance:
uuid: "376a6ab84b28482094a9eb924d11e5f7"
format_stamp: "Formatted at 2026-08-07 23:01:39 on dist-test-slave-z8x9"
server_key: "5e5aa8b22f379d9d0dcc697014c15c0c"
server_key_iv: "7472a91d1bbe3386b545651f3e1856bf"
server_key_version: "encryptionkey@0"
I20260807 23:01:39.572669 25777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786143361779964-25976-0/minicluster-data/ts-0/wal/instance:
uuid: "376a6ab84b28482094a9eb924d11e5f7"
format_stamp: "Formatted at 2026-08-07 23:01:39 on dist-test-slave-z8x9"
server_key: "5e5aa8b22f379d9d0dcc697014c15c0c"
server_key_iv: "7472a91d1bbe3386b545651f3e1856bf"
server_key_version: "encryptionkey@0"
I20260807 23:01:39.578343 25777 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260807 23:01:39.582508 25796 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 23:01:39.584071 25777 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260807 23:01:39.584290 25777 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786143361779964-25976-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786143361779964-25976-0/minicluster-data/ts-0/wal
uuid: "376a6ab84b28482094a9eb924d11e5f7"
format_stamp: "Formatted at 2026-08-07 23:01:39 on dist-test-slave-z8x9"
server_key: "5e5aa8b22f379d9d0dcc697014c15c0c"
server_key_iv: "7472a91d1bbe3386b545651f3e1856bf"
server_key_version: "encryptionkey@0"
I20260807 23:01:39.584528 25777 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786143361779964-25976-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786143361779964-25976-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786143361779964-25976-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 23:01:39.598826 25777 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 23:01:39.603382 25777 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 23:01:39.603703 25777 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 23:01:39.604743 25777 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 23:01:39.606124 25777 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 23:01:39.606230 25777 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 23:01:39.606396 25777 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 23:01:39.606477 25777 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 23:01:39.641659 25777 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.1:45303
I20260807 23:01:39.641763 25970 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.1:45303 every 8 connection(s)
I20260807 23:01:39.643054 25777 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786143361779964-25976-0/minicluster-data/ts-0/data/info.pb
I20260807 23:01:39.645609 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 25777
I20260807 23:01:39.645838 25976 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786143361779964-25976-0/minicluster-data/ts-0/wal/instance
I20260807 23:01:39.646401 25976 external_mini_cluster.cc:1428] Setting key 74708298051db7b727e6435a3eeb7626
Aug 07 23:01:39 dist-test-slave-z8x9 krb5kdc[25610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143699, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.1@KRBTEST.COM for kudu/127.25.94.62@KRBTEST.COM
I20260807 23:01:39.665980 25973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:39.645817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.1:58201 (local address 127.25.94.62:36775)
0807 23:01:39.646341 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0807 23:01:39.646347 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 23:01:39.647349 (+ 1002us) server_negotiation.cc:407] Connection header received
0807 23:01:39.649841 (+ 2492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:01:39.649845 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:01:39.649911 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:01:39.649994 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:01:39.652102 (+ 2108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:39.652835 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:39.655331 (+ 2496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:39.655703 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:39.655811 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:39.660134 (+ 4323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:01:39.660147 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:01:39.660151 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 23:01:39.660184 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:01:39.662872 (+ 2688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:01:39.663619 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:01:39.663624 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:01:39.663629 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:01:39.663759 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:01:39.664200 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:01:39.664204 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:01:39.664206 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:01:39.664460 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:01:39.664611 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0807 23:01:39.665343 (+ 732us) server_negotiation.cc:299] Negotiation successful
0807 23:01:39.665583 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":247,"threads_started":1}
I20260807 23:01:39.666939 25974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:39.646331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:36775 (local address 127.25.94.1:58201)
0807 23:01:39.647175 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0807 23:01:39.647212 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0807 23:01:39.649537 (+ 2325us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 23:01:39.650279 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 23:01:39.650292 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 23:01:39.651099 (+ 807us) client_negotiation.cc:189] Negotiated authn=SASL
0807 23:01:39.651927 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:01:39.651957 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:01:39.653015 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:01:39.653023 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:01:39.655167 (+ 2144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:01:39.655185 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:01:39.655939 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:01:39.655948 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:01:39.656293 (+ 345us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:39.657094 (+ 801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 23:01:39.657128 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0807 23:01:39.659925 (+ 2797us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 23:01:39.663026 (+ 3101us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 23:01:39.663035 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 23:01:39.663051 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0807 23:01:39.663490 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 23:01:39.663869 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 23:01:39.663875 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 23:01:39.663878 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 23:01:39.664071 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 23:01:39.664599 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 23:01:39.664607 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 23:01:39.664946 (+ 339us) client_negotiation.cc:769] Sending connection context
0807 23:01:39.665220 (+ 274us) client_negotiation.cc:240] Negotiation successful
0807 23:01:39.665774 (+ 554us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":339,"threads_started":1}
I20260807 23:01:39.668668 25971 heartbeater.cc:347] Connected to a master server at 127.25.94.62:36775
I20260807 23:01:39.669049 25971 heartbeater.cc:464] Registering TS with master...
I20260807 23:01:39.670051 25971 heartbeater.cc:511] Master 127.25.94.62:36775 requested a full tablet report, sending...
I20260807 23:01:39.673087 25685 ts_manager.cc:194] Registered new tserver with Master: 376a6ab84b28482094a9eb924d11e5f7 (127.25.94.1:45303)
I20260807 23:01:39.675403 25685 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.94.1@KRBTEST.COM'} at 127.25.94.1:58201
WARNING: no policy specified for kudu/127.25.94.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.2.keytab.
Entry for principal kudu/127.25.94.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.2.keytab.
I20260807 23:01:39.721309 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786143361779964-25976-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786143361779964-25976-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786143361779964-25976-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786143361779964-25976-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.2:0
--local_ip_for_outbound_sockets=127.25.94.2
--webserver_interface=127.25.94.2
--webserver_port=0
--tserver_master_addrs=127.25.94.62:36775
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.2.keytab
--principal=kudu/127.25.94.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.20:32943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786143361779964-25976-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786143361779964-25976-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786143361779964-25976-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 23:01:40.064395 25987 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 23:01:40.064702 25987 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 23:01:40.064801 25987 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 23:01:40.072250 25987 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 23:01:40.072382 25987 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 23:01:40.072499 25987 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.2
I20260807 23:01:40.083060 25987 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:32943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786143361779964-25976-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786143361779964-25976-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.2.keytab
--principal=kudu/127.25.94.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786143361779964-25976-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.94.2
--webserver_port=0
--tserver_master_addrs=127.25.94.62:36775
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.2
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786143361779964-25976-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 23:01:40.084376 25987 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 23:01:40.085861 25987 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 23:01:40.101230 25993 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 23:01:40.101262 25994 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 23:01:40.101207 25998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 23:01:40.102313 25987 server_base.cc:1034] running on GCE node
I20260807 23:01:40.103122 25987 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 23:01:40.104203 25987 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 23:01:40.105434 25987 hybrid_clock.cc:648] HybridClock initialized: now 1786143700105380 us; error 46 us; skew 500 ppm
I20260807 23:01:40.105816 25987 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 23:01:40 dist-test-slave-z8x9 krb5kdc[25610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143700, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 23:01:40.109485 25987 init.cc:377] Logged in from keytab as kudu/127.25.94.2@KRBTEST.COM (short username kudu)
I20260807 23:01:40.111680 25987 webserver.cc:460] Webserver started at http://127.25.94.2:42655/ using document root <none> and password file <none>
I20260807 23:01:40.112807 25987 fs_manager.cc:359] Metadata directory not provided
I20260807 23:01:40.113015 25987 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 23:01:40.113487 25987 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 23:01:40.116725 25987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786143361779964-25976-0/minicluster-data/ts-1/data/instance:
uuid: "5cc8d20073804cc2b0e6f22f2ea059cd"
format_stamp: "Formatted at 2026-08-07 23:01:40 on dist-test-slave-z8x9"
server_key: "db9d3ecb2e50efe3c84b2216706e820c"
server_key_iv: "3ff808a2724e511ab9ebeae475bc2624"
server_key_version: "encryptionkey@0"
I20260807 23:01:40.117690 25987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786143361779964-25976-0/minicluster-data/ts-1/wal/instance:
uuid: "5cc8d20073804cc2b0e6f22f2ea059cd"
format_stamp: "Formatted at 2026-08-07 23:01:40 on dist-test-slave-z8x9"
server_key: "db9d3ecb2e50efe3c84b2216706e820c"
server_key_iv: "3ff808a2724e511ab9ebeae475bc2624"
server_key_version: "encryptionkey@0"
I20260807 23:01:40.124125 25987 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260807 23:01:40.128541 26010 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 23:01:40.130196 25987 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260807 23:01:40.130435 25987 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786143361779964-25976-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786143361779964-25976-0/minicluster-data/ts-1/wal
uuid: "5cc8d20073804cc2b0e6f22f2ea059cd"
format_stamp: "Formatted at 2026-08-07 23:01:40 on dist-test-slave-z8x9"
server_key: "db9d3ecb2e50efe3c84b2216706e820c"
server_key_iv: "3ff808a2724e511ab9ebeae475bc2624"
server_key_version: "encryptionkey@0"
I20260807 23:01:40.130699 25987 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786143361779964-25976-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786143361779964-25976-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786143361779964-25976-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 23:01:40.154203 25987 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 23:01:40.160145 25987 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 23:01:40.160563 25987 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 23:01:40.161864 25987 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 23:01:40.163519 25987 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 23:01:40.163637 25987 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 23:01:40.163801 25987 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 23:01:40.163879 25987 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 23:01:40.204771 25987 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.2:36545
I20260807 23:01:40.204826 26197 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.2:36545 every 8 connection(s)
I20260807 23:01:40.206301 25987 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786143361779964-25976-0/minicluster-data/ts-1/data/info.pb
I20260807 23:01:40.214134 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 25987
I20260807 23:01:40.214514 25976 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786143361779964-25976-0/minicluster-data/ts-1/wal/instance
I20260807 23:01:40.215277 25976 external_mini_cluster.cc:1428] Setting key f1b714e1047ac5c9e261083c5a44a826
Aug 07 23:01:40 dist-test-slave-z8x9 krb5kdc[25610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143700, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.2@KRBTEST.COM for kudu/127.25.94.62@KRBTEST.COM
I20260807 23:01:40.232787 26202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:40.209240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.2:53157 (local address 127.25.94.62:36775)
0807 23:01:40.209757 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0807 23:01:40.209762 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 23:01:40.210649 (+ 887us) server_negotiation.cc:407] Connection header received
0807 23:01:40.212865 (+ 2216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:01:40.212871 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:01:40.212975 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:01:40.213104 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:01:40.215209 (+ 2105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:40.216172 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:40.219212 (+ 3040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:40.219776 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:40.219892 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:40.224971 (+ 5079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:01:40.224977 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:01:40.224981 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 23:01:40.225018 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:01:40.228084 (+ 3066us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:01:40.229340 (+ 1256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:01:40.229345 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:01:40.229347 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:01:40.229515 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:01:40.230141 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:01:40.230146 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:01:40.230148 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:01:40.230645 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:01:40.230791 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0807 23:01:40.231933 (+ 1142us) server_negotiation.cc:299] Negotiation successful
0807 23:01:40.232294 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":266,"threads_started":1}
I20260807 23:01:40.233551 26204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:40.209666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:36775 (local address 127.25.94.2:53157)
0807 23:01:40.210479 (+ 813us) negotiation.cc:107] Waiting for socket to connect
0807 23:01:40.210517 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0807 23:01:40.212548 (+ 2031us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 23:01:40.213420 (+ 872us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 23:01:40.213429 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 23:01:40.214052 (+ 623us) client_negotiation.cc:189] Negotiated authn=SASL
0807 23:01:40.214956 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:01:40.214984 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:01:40.216883 (+ 1899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:01:40.216888 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:01:40.219041 (+ 2153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:01:40.219065 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:01:40.220037 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:01:40.220049 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:01:40.220424 (+ 375us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:40.221311 (+ 887us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 23:01:40.221348 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0807 23:01:40.224621 (+ 3273us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 23:01:40.228290 (+ 3669us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 23:01:40.228303 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 23:01:40.228327 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0807 23:01:40.229057 (+ 730us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 23:01:40.229639 (+ 582us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 23:01:40.229646 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 23:01:40.229652 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 23:01:40.229938 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 23:01:40.230802 (+ 864us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 23:01:40.230813 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 23:01:40.231343 (+ 530us) client_negotiation.cc:769] Sending connection context
0807 23:01:40.231693 (+ 350us) client_negotiation.cc:240] Negotiation successful
0807 23:01:40.232237 (+ 544us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":296,"threads_started":1}
I20260807 23:01:40.235232 26198 heartbeater.cc:347] Connected to a master server at 127.25.94.62:36775
I20260807 23:01:40.235608 26198 heartbeater.cc:464] Registering TS with master...
I20260807 23:01:40.236605 26198 heartbeater.cc:511] Master 127.25.94.62:36775 requested a full tablet report, sending...
I20260807 23:01:40.238561 25685 ts_manager.cc:194] Registered new tserver with Master: 5cc8d20073804cc2b0e6f22f2ea059cd (127.25.94.2:36545)
I20260807 23:01:40.239522 25685 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.94.2@KRBTEST.COM'} at 127.25.94.2:53157
WARNING: no policy specified for kudu/127.25.94.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.3.keytab.
Entry for principal kudu/127.25.94.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.3.keytab.
I20260807 23:01:40.293546 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786143361779964-25976-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786143361779964-25976-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786143361779964-25976-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786143361779964-25976-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.3:0
--local_ip_for_outbound_sockets=127.25.94.3
--webserver_interface=127.25.94.3
--webserver_port=0
--tserver_master_addrs=127.25.94.62:36775
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.3.keytab
--principal=kudu/127.25.94.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.20:32943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786143361779964-25976-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786143361779964-25976-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786143361779964-25976-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 23:01:40.651885 26208 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 23:01:40.652184 26208 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 23:01:40.652371 26208 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 23:01:40.660025 26208 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 23:01:40.660212 26208 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 23:01:40.660411 26208 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.3
I20260807 23:01:40.672788 26208 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:32943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786143361779964-25976-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786143361779964-25976-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.3.keytab
--principal=kudu/127.25.94.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786143361779964-25976-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.94.3
--webserver_port=0
--tserver_master_addrs=127.25.94.62:36775
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.3
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786143361779964-25976-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 23:01:40.674217 26208 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 23:01:40.675630 26208 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260807 23:01:40.679798 25971 heartbeater.cc:503] Master 127.25.94.62:36775 was elected leader, sending a full tablet report...
W20260807 23:01:40.690006 26214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 23:01:40.690044 26218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 23:01:40.690084 26216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 23:01:40.691434 26208 server_base.cc:1034] running on GCE node
I20260807 23:01:40.692165 26208 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 23:01:40.693248 26208 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 23:01:40.694507 26208 hybrid_clock.cc:648] HybridClock initialized: now 1786143700694434 us; error 59 us; skew 500 ppm
I20260807 23:01:40.695010 26208 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 23:01:40 dist-test-slave-z8x9 krb5kdc[25610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143700, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 23:01:40.698602 26208 init.cc:377] Logged in from keytab as kudu/127.25.94.3@KRBTEST.COM (short username kudu)
I20260807 23:01:40.700232 26208 webserver.cc:460] Webserver started at http://127.25.94.3:34667/ using document root <none> and password file <none>
I20260807 23:01:40.700973 26208 fs_manager.cc:359] Metadata directory not provided
I20260807 23:01:40.701109 26208 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 23:01:40.701467 26208 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 23:01:40.704372 26208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786143361779964-25976-0/minicluster-data/ts-2/data/instance:
uuid: "31051d460d394d6da8385c2f82b3ab7e"
format_stamp: "Formatted at 2026-08-07 23:01:40 on dist-test-slave-z8x9"
server_key: "0bfcb1cc94a99e13df96cdf20318f353"
server_key_iv: "f02af33df57eb2cad2a6ede283d4c65d"
server_key_version: "encryptionkey@0"
I20260807 23:01:40.705226 26208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786143361779964-25976-0/minicluster-data/ts-2/wal/instance:
uuid: "31051d460d394d6da8385c2f82b3ab7e"
format_stamp: "Formatted at 2026-08-07 23:01:40 on dist-test-slave-z8x9"
server_key: "0bfcb1cc94a99e13df96cdf20318f353"
server_key_iv: "f02af33df57eb2cad2a6ede283d4c65d"
server_key_version: "encryptionkey@0"
I20260807 23:01:40.710537 26208 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260807 23:01:40.714401 26232 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 23:01:40.715750 26208 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 23:01:40.715929 26208 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786143361779964-25976-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786143361779964-25976-0/minicluster-data/ts-2/wal
uuid: "31051d460d394d6da8385c2f82b3ab7e"
format_stamp: "Formatted at 2026-08-07 23:01:40 on dist-test-slave-z8x9"
server_key: "0bfcb1cc94a99e13df96cdf20318f353"
server_key_iv: "f02af33df57eb2cad2a6ede283d4c65d"
server_key_version: "encryptionkey@0"
I20260807 23:01:40.716140 26208 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786143361779964-25976-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786143361779964-25976-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786143361779964-25976-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 23:01:40.736529 26208 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 23:01:40.740510 26208 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 23:01:40.740808 26208 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 23:01:40.741866 26208 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 23:01:40.743252 26208 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 23:01:40.743361 26208 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 23:01:40.743512 26208 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 23:01:40.743587 26208 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 23:01:40.779752 26208 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.3:46437
I20260807 23:01:40.779784 26422 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.3:46437 every 8 connection(s)
I20260807 23:01:40.781045 26208 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786143361779964-25976-0/minicluster-data/ts-2/data/info.pb
I20260807 23:01:40.786605 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 26208
I20260807 23:01:40.786808 25976 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786143361779964-25976-0/minicluster-data/ts-2/wal/instance
I20260807 23:01:40.787293 25976 external_mini_cluster.cc:1428] Setting key 21d69be6be83b439f5bce7d82932d979
Aug 07 23:01:40 dist-test-slave-z8x9 krb5kdc[25610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143700, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.3@KRBTEST.COM for kudu/127.25.94.62@KRBTEST.COM
I20260807 23:01:40.802840 26426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:40.783718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.3:55807 (local address 127.25.94.62:36775)
0807 23:01:40.784229 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0807 23:01:40.784233 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 23:01:40.785008 (+ 775us) server_negotiation.cc:407] Connection header received
0807 23:01:40.787049 (+ 2041us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:01:40.787055 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:01:40.787152 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:01:40.787273 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:01:40.789376 (+ 2103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:40.790338 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:40.791899 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:40.792309 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:40.792448 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:40.796143 (+ 3695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:01:40.796154 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:01:40.796158 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 23:01:40.796193 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:01:40.799133 (+ 2940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:01:40.799946 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:01:40.799952 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:01:40.799955 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:01:40.800090 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:01:40.800575 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:01:40.800580 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:01:40.800582 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:01:40.800924 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:01:40.801104 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0807 23:01:40.802210 (+ 1106us) server_negotiation.cc:299] Negotiation successful
0807 23:01:40.802511 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":260,"threads_started":1}
I20260807 23:01:40.803958 26427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:40.784037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:36775 (local address 127.25.94.3:55807)
0807 23:01:40.784837 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0807 23:01:40.784880 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0807 23:01:40.786719 (+ 1839us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 23:01:40.787515 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 23:01:40.787523 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 23:01:40.788295 (+ 772us) client_negotiation.cc:189] Negotiated authn=SASL
0807 23:01:40.789100 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:01:40.789133 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:01:40.790492 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:01:40.790498 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:01:40.791730 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:01:40.791743 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:01:40.792549 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:01:40.792559 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:01:40.792806 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:40.793611 (+ 805us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 23:01:40.793639 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0807 23:01:40.795892 (+ 2253us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 23:01:40.799306 (+ 3414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 23:01:40.799315 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 23:01:40.799335 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0807 23:01:40.799805 (+ 470us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 23:01:40.800211 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 23:01:40.800216 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 23:01:40.800220 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 23:01:40.800432 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 23:01:40.801264 (+ 832us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 23:01:40.801273 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 23:01:40.801759 (+ 486us) client_negotiation.cc:769] Sending connection context
0807 23:01:40.802087 (+ 328us) client_negotiation.cc:240] Negotiation successful
0807 23:01:40.802679 (+ 592us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":284,"threads_started":1}
I20260807 23:01:40.805625 26423 heartbeater.cc:347] Connected to a master server at 127.25.94.62:36775
I20260807 23:01:40.805980 26423 heartbeater.cc:464] Registering TS with master...
I20260807 23:01:40.806756 26423 heartbeater.cc:511] Master 127.25.94.62:36775 requested a full tablet report, sending...
I20260807 23:01:40.808204 25685 ts_manager.cc:194] Registered new tserver with Master: 31051d460d394d6da8385c2f82b3ab7e (127.25.94.3:46437)
I20260807 23:01:40.809034 25685 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.94.3@KRBTEST.COM'} at 127.25.94.3:55807
I20260807 23:01:40.815579 25976 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 23:01:40.832333 26426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:40.819722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37048 (local address 127.25.94.62:36775)
0807 23:01:40.819886 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0807 23:01:40.819891 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 23:01:40.820401 (+ 510us) server_negotiation.cc:407] Connection header received
0807 23:01:40.820555 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:01:40.820559 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:01:40.820622 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:01:40.820719 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:01:40.822278 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:40.822985 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:40.824193 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:40.824617 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:40.824711 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:40.827026 (+ 2315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:01:40.827031 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:01:40.827035 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 23:01:40.827067 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:01:40.829836 (+ 2769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:01:40.830532 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:01:40.830537 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:01:40.830539 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:01:40.830669 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:01:40.831022 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:01:40.831025 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:01:40.831027 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:01:40.831373 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:01:40.831565 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0807 23:01:40.831903 (+ 338us) server_negotiation.cc:299] Negotiation successful
0807 23:01:40.832075 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260807 23:01:40.836692 25685 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37048:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 23:01:40.843927 25685 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 23:01:40.873494 26445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:40.856843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.3:46437 (local address 127.0.0.1:58226)
0807 23:01:40.858641 (+ 1798us) negotiation.cc:107] Waiting for socket to connect
0807 23:01:40.858687 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0807 23:01:40.858859 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 23:01:40.861314 (+ 2455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 23:01:40.861326 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 23:01:40.861369 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 23:01:40.861836 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:01:40.861852 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:01:40.865631 (+ 3779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:01:40.865640 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:01:40.869977 (+ 4337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:01:40.870003 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:01:40.872321 (+ 2318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:01:40.872329 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:01:40.872587 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:40.872663 (+ 76us) client_negotiation.cc:769] Sending connection context
0807 23:01:40.872808 (+ 145us) client_negotiation.cc:240] Negotiation successful
0807 23:01:40.873042 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1569,"thread_start_us":1425,"threads_started":1}
I20260807 23:01:40.873574 26446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:40.857095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58226 (local address 127.25.94.3:46437)
0807 23:01:40.857891 (+ 796us) server_negotiation.cc:206] Beginning negotiation
0807 23:01:40.857902 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 23:01:40.859065 (+ 1163us) server_negotiation.cc:407] Connection header received
0807 23:01:40.859199 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:01:40.859216 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:01:40.859479 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:01:40.859702 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 23:01:40.862093 (+ 2391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:40.865393 (+ 3300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:40.870259 (+ 4866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:40.872087 (+ 1828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:40.872232 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:40.872479 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0807 23:01:40.873037 (+ 558us) server_negotiation.cc:299] Negotiation successful
0807 23:01:40.873237 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":591,"thread_start_us":359,"threads_started":1}
I20260807 23:01:40.878060 26313 tablet_service.cc:1467] Processing CreateTablet for tablet 93ca732f90794f3890e4ab071ee4d10d (DEFAULT_TABLE table=test-table [id=7e7e86e919c74415997d2d897339342b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 23:01:40.879554 26313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93ca732f90794f3890e4ab071ee4d10d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 23:01:40.880307 26450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:40.859098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.1:45303 (local address 127.0.0.1:39788)
0807 23:01:40.871924 (+ 12826us) negotiation.cc:107] Waiting for socket to connect
0807 23:01:40.871948 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0807 23:01:40.872073 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 23:01:40.872663 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 23:01:40.872668 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 23:01:40.872698 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 23:01:40.873138 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:01:40.873163 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:01:40.875421 (+ 2258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:01:40.875426 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:01:40.877514 (+ 2088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:01:40.877530 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:01:40.879475 (+ 1945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:01:40.879482 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:01:40.879685 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:40.879720 (+ 35us) client_negotiation.cc:769] Sending connection context
0807 23:01:40.879811 (+ 91us) client_negotiation.cc:240] Negotiation successful
0807 23:01:40.879946 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12715,"thread_start_us":10685,"threads_started":1}
I20260807 23:01:40.880427 26452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:40.859759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39788 (local address 127.25.94.1:45303)
0807 23:01:40.871102 (+ 11343us) server_negotiation.cc:206] Beginning negotiation
0807 23:01:40.871116 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0807 23:01:40.872074 (+ 958us) server_negotiation.cc:407] Connection header received
0807 23:01:40.872217 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:01:40.872225 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:01:40.872443 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:01:40.872687 (+ 244us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 23:01:40.873993 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:40.875274 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:40.877713 (+ 2439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:40.879175 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:40.879298 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:40.879531 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0807 23:01:40.879900 (+ 369us) server_negotiation.cc:299] Negotiation successful
0807 23:01:40.880082 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11102,"thread_start_us":10908,"threads_started":1}
I20260807 23:01:40.883296 26451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:40.857892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.2:36545 (local address 127.0.0.1:46760)
0807 23:01:40.870184 (+ 12292us) negotiation.cc:107] Waiting for socket to connect
0807 23:01:40.870210 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0807 23:01:40.870368 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 23:01:40.872267 (+ 1899us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 23:01:40.872274 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 23:01:40.872307 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 23:01:40.872781 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:01:40.872795 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:01:40.877420 (+ 4625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:01:40.877432 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:01:40.880697 (+ 3265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:01:40.880714 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:01:40.882377 (+ 1663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:01:40.882384 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:01:40.882558 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:40.882597 (+ 39us) client_negotiation.cc:769] Sending connection context
0807 23:01:40.882711 (+ 114us) client_negotiation.cc:240] Negotiation successful
0807 23:01:40.882844 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12092,"thread_start_us":13760,"threads_started":1}
I20260807 23:01:40.883386 26447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:40.857592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46760 (local address 127.25.94.2:36545)
0807 23:01:40.871565 (+ 13973us) server_negotiation.cc:206] Beginning negotiation
0807 23:01:40.871577 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 23:01:40.871615 (+ 38us) server_negotiation.cc:407] Connection header received
0807 23:01:40.871726 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:01:40.871735 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:01:40.872045 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:01:40.873006 (+ 961us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 23:01:40.873266 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:40.877274 (+ 4008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:40.880903 (+ 3629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:40.882152 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:40.882312 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:40.882510 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0807 23:01:40.882766 (+ 256us) server_negotiation.cc:299] Negotiation successful
0807 23:01:40.882925 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13817,"thread_start_us":13723,"threads_started":1}
I20260807 23:01:40.884680 25863 tablet_service.cc:1467] Processing CreateTablet for tablet 93ca732f90794f3890e4ab071ee4d10d (DEFAULT_TABLE table=test-table [id=7e7e86e919c74415997d2d897339342b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 23:01:40.886386 25863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93ca732f90794f3890e4ab071ee4d10d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 23:01:40.886927 26083 tablet_service.cc:1467] Processing CreateTablet for tablet 93ca732f90794f3890e4ab071ee4d10d (DEFAULT_TABLE table=test-table [id=7e7e86e919c74415997d2d897339342b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 23:01:40.888042 26083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93ca732f90794f3890e4ab071ee4d10d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 23:01:40.895200 26454 tablet_bootstrap.cc:492] T 93ca732f90794f3890e4ab071ee4d10d P 31051d460d394d6da8385c2f82b3ab7e: Bootstrap starting.
I20260807 23:01:40.899087 26454 tablet_bootstrap.cc:654] T 93ca732f90794f3890e4ab071ee4d10d P 31051d460d394d6da8385c2f82b3ab7e: Neither blocks nor log segments found. Creating new log.
I20260807 23:01:40.900424 26454 log.cc:824] T 93ca732f90794f3890e4ab071ee4d10d P 31051d460d394d6da8385c2f82b3ab7e: Log is configured to *not* fsync() on all Append() calls
I20260807 23:01:40.900946 26457 tablet_bootstrap.cc:492] T 93ca732f90794f3890e4ab071ee4d10d P 376a6ab84b28482094a9eb924d11e5f7: Bootstrap starting.
I20260807 23:01:40.902523 26459 tablet_bootstrap.cc:492] T 93ca732f90794f3890e4ab071ee4d10d P 5cc8d20073804cc2b0e6f22f2ea059cd: Bootstrap starting.
I20260807 23:01:40.905016 26457 tablet_bootstrap.cc:654] T 93ca732f90794f3890e4ab071ee4d10d P 376a6ab84b28482094a9eb924d11e5f7: Neither blocks nor log segments found. Creating new log.
I20260807 23:01:40.905647 26454 tablet_bootstrap.cc:492] T 93ca732f90794f3890e4ab071ee4d10d P 31051d460d394d6da8385c2f82b3ab7e: No bootstrap required, opened a new log
I20260807 23:01:40.905937 26454 ts_tablet_manager.cc:1397] T 93ca732f90794f3890e4ab071ee4d10d P 31051d460d394d6da8385c2f82b3ab7e: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.003s
I20260807 23:01:40.906157 26459 tablet_bootstrap.cc:654] T 93ca732f90794f3890e4ab071ee4d10d P 5cc8d20073804cc2b0e6f22f2ea059cd: Neither blocks nor log segments found. Creating new log.
I20260807 23:01:40.906333 26457 log.cc:824] T 93ca732f90794f3890e4ab071ee4d10d P 376a6ab84b28482094a9eb924d11e5f7: Log is configured to *not* fsync() on all Append() calls
I20260807 23:01:40.907348 26459 log.cc:824] T 93ca732f90794f3890e4ab071ee4d10d P 5cc8d20073804cc2b0e6f22f2ea059cd: Log is configured to *not* fsync() on all Append() calls
I20260807 23:01:40.910316 26459 tablet_bootstrap.cc:492] T 93ca732f90794f3890e4ab071ee4d10d P 5cc8d20073804cc2b0e6f22f2ea059cd: No bootstrap required, opened a new log
I20260807 23:01:40.910351 26457 tablet_bootstrap.cc:492] T 93ca732f90794f3890e4ab071ee4d10d P 376a6ab84b28482094a9eb924d11e5f7: No bootstrap required, opened a new log
I20260807 23:01:40.910732 26459 ts_tablet_manager.cc:1397] T 93ca732f90794f3890e4ab071ee4d10d P 5cc8d20073804cc2b0e6f22f2ea059cd: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260807 23:01:40.910748 26457 ts_tablet_manager.cc:1397] T 93ca732f90794f3890e4ab071ee4d10d P 376a6ab84b28482094a9eb924d11e5f7: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.002s
I20260807 23:01:40.914969 26454 raft_consensus.cc:348] T 93ca732f90794f3890e4ab071ee4d10d P 31051d460d394d6da8385c2f82b3ab7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31051d460d394d6da8385c2f82b3ab7e" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 46437 } } peers { permanent_uuid: "376a6ab84b28482094a9eb924d11e5f7" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 45303 } } peers { permanent_uuid: "5cc8d20073804cc2b0e6f22f2ea059cd" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 36545 } }
I20260807 23:01:40.915343 26454 raft_consensus.cc:374] T 93ca732f90794f3890e4ab071ee4d10d P 31051d460d394d6da8385c2f82b3ab7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 23:01:40.915479 26454 raft_consensus.cc:729] T 93ca732f90794f3890e4ab071ee4d10d P 31051d460d394d6da8385c2f82b3ab7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31051d460d394d6da8385c2f82b3ab7e, State: Initialized, Role: FOLLOWER
I20260807 23:01:40.916116 26454 consensus_queue.cc:260] T 93ca732f90794f3890e4ab071ee4d10d P 31051d460d394d6da8385c2f82b3ab7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31051d460d394d6da8385c2f82b3ab7e" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 46437 } } peers { permanent_uuid: "376a6ab84b28482094a9eb924d11e5f7" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 45303 } } peers { permanent_uuid: "5cc8d20073804cc2b0e6f22f2ea059cd" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 36545 } }
I20260807 23:01:40.917315 26423 heartbeater.cc:503] Master 127.25.94.62:36775 was elected leader, sending a full tablet report...
I20260807 23:01:40.918010 26454 ts_tablet_manager.cc:1428] T 93ca732f90794f3890e4ab071ee4d10d P 31051d460d394d6da8385c2f82b3ab7e: Time spent starting tablet: real 0.012s user 0.011s sys 0.001s
I20260807 23:01:40.921459 26457 raft_consensus.cc:348] T 93ca732f90794f3890e4ab071ee4d10d P 376a6ab84b28482094a9eb924d11e5f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31051d460d394d6da8385c2f82b3ab7e" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 46437 } } peers { permanent_uuid: "376a6ab84b28482094a9eb924d11e5f7" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 45303 } } peers { permanent_uuid: "5cc8d20073804cc2b0e6f22f2ea059cd" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 36545 } }
I20260807 23:01:40.921999 26457 raft_consensus.cc:374] T 93ca732f90794f3890e4ab071ee4d10d P 376a6ab84b28482094a9eb924d11e5f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 23:01:40.922160 26457 raft_consensus.cc:729] T 93ca732f90794f3890e4ab071ee4d10d P 376a6ab84b28482094a9eb924d11e5f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 376a6ab84b28482094a9eb924d11e5f7, State: Initialized, Role: FOLLOWER
I20260807 23:01:40.922825 26457 consensus_queue.cc:260] T 93ca732f90794f3890e4ab071ee4d10d P 376a6ab84b28482094a9eb924d11e5f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31051d460d394d6da8385c2f82b3ab7e" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 46437 } } peers { permanent_uuid: "376a6ab84b28482094a9eb924d11e5f7" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 45303 } } peers { permanent_uuid: "5cc8d20073804cc2b0e6f22f2ea059cd" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 36545 } }
I20260807 23:01:40.922803 26459 raft_consensus.cc:348] T 93ca732f90794f3890e4ab071ee4d10d P 5cc8d20073804cc2b0e6f22f2ea059cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31051d460d394d6da8385c2f82b3ab7e" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 46437 } } peers { permanent_uuid: "376a6ab84b28482094a9eb924d11e5f7" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 45303 } } peers { permanent_uuid: "5cc8d20073804cc2b0e6f22f2ea059cd" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 36545 } }
I20260807 23:01:40.923159 26459 raft_consensus.cc:374] T 93ca732f90794f3890e4ab071ee4d10d P 5cc8d20073804cc2b0e6f22f2ea059cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 23:01:40.923269 26459 raft_consensus.cc:729] T 93ca732f90794f3890e4ab071ee4d10d P 5cc8d20073804cc2b0e6f22f2ea059cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cc8d20073804cc2b0e6f22f2ea059cd, State: Initialized, Role: FOLLOWER
I20260807 23:01:40.923987 26459 consensus_queue.cc:260] T 93ca732f90794f3890e4ab071ee4d10d P 5cc8d20073804cc2b0e6f22f2ea059cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31051d460d394d6da8385c2f82b3ab7e" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 46437 } } peers { permanent_uuid: "376a6ab84b28482094a9eb924d11e5f7" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 45303 } } peers { permanent_uuid: "5cc8d20073804cc2b0e6f22f2ea059cd" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 36545 } }
I20260807 23:01:40.924630 26457 ts_tablet_manager.cc:1428] T 93ca732f90794f3890e4ab071ee4d10d P 376a6ab84b28482094a9eb924d11e5f7: Time spent starting tablet: real 0.014s user 0.010s sys 0.002s
I20260807 23:01:40.925007 26198 heartbeater.cc:503] Master 127.25.94.62:36775 was elected leader, sending a full tablet report...
I20260807 23:01:40.925678 26459 ts_tablet_manager.cc:1428] T 93ca732f90794f3890e4ab071ee4d10d P 5cc8d20073804cc2b0e6f22f2ea059cd: Time spent starting tablet: real 0.015s user 0.012s sys 0.001s
W20260807 23:01:40.958542 26200 tablet.cc:2369] T 93ca732f90794f3890e4ab071ee4d10d P 5cc8d20073804cc2b0e6f22f2ea059cd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 23:01:41.032809 26425 tablet.cc:2369] T 93ca732f90794f3890e4ab071ee4d10d P 31051d460d394d6da8385c2f82b3ab7e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 23:01:41.144392 26464 raft_consensus.cc:482] T 93ca732f90794f3890e4ab071ee4d10d P 31051d460d394d6da8385c2f82b3ab7e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 23:01:41.144862 26464 raft_consensus.cc:504] T 93ca732f90794f3890e4ab071ee4d10d P 31051d460d394d6da8385c2f82b3ab7e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31051d460d394d6da8385c2f82b3ab7e" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 46437 } } peers { permanent_uuid: "376a6ab84b28482094a9eb924d11e5f7" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 45303 } } peers { permanent_uuid: "5cc8d20073804cc2b0e6f22f2ea059cd" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 36545 } }
W20260807 23:01:41.145675 25972 tablet.cc:2369] T 93ca732f90794f3890e4ab071ee4d10d P 376a6ab84b28482094a9eb924d11e5f7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 23:01:41.146780 26464 leader_election.cc:288] T 93ca732f90794f3890e4ab071ee4d10d P 31051d460d394d6da8385c2f82b3ab7e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 376a6ab84b28482094a9eb924d11e5f7 (127.25.94.1:45303), 5cc8d20073804cc2b0e6f22f2ea059cd (127.25.94.2:36545)
I20260807 23:01:41.154033 26447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:41.147275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.3:46859 (local address 127.25.94.2:36545)
0807 23:01:41.147484 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0807 23:01:41.147491 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 23:01:41.147704 (+ 213us) server_negotiation.cc:407] Connection header received
0807 23:01:41.147995 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:01:41.148004 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:01:41.148099 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:01:41.148276 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 23:01:41.148828 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:41.149893 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:41.151865 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:41.153072 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:41.153247 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:41.153345 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0807 23:01:41.153646 (+ 301us) server_negotiation.cc:299] Negotiation successful
0807 23:01:41.153769 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260807 23:01:41.154043 26427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:41.147145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.2:36545 (local address 127.25.94.3:46859)
0807 23:01:41.147550 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0807 23:01:41.147579 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0807 23:01:41.147749 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 23:01:41.148304 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 23:01:41.148307 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 23:01:41.148331 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 23:01:41.148578 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:01:41.148590 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:01:41.150049 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:01:41.150056 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:01:41.151680 (+ 1624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:01:41.151694 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:01:41.153257 (+ 1563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:01:41.153263 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:01:41.153460 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:41.153496 (+ 36us) client_negotiation.cc:769] Sending connection context
0807 23:01:41.153606 (+ 110us) client_negotiation.cc:240] Negotiation successful
0807 23:01:41.153746 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250}
I20260807 23:01:41.155196 26452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:41.147275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.3:58057 (local address 127.25.94.1:45303)
0807 23:01:41.147484 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0807 23:01:41.147490 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 23:01:41.148442 (+ 952us) server_negotiation.cc:407] Connection header received
0807 23:01:41.148621 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:01:41.148629 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:01:41.148728 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:01:41.148804 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 23:01:41.149619 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:41.150673 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:41.152898 (+ 2225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:41.154143 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:41.154303 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:41.154397 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0807 23:01:41.154801 (+ 404us) server_negotiation.cc:299] Negotiation successful
0807 23:01:41.154964 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260807 23:01:41.155211 26467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:41.147325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.1:45303 (local address 127.25.94.3:58057)
0807 23:01:41.148316 (+ 991us) negotiation.cc:107] Waiting for socket to connect
0807 23:01:41.148338 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 23:01:41.148480 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 23:01:41.148884 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 23:01:41.148893 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 23:01:41.148926 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 23:01:41.149394 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:01:41.149409 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:01:41.150849 (+ 1440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:01:41.150853 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:01:41.152648 (+ 1795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:01:41.152664 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:01:41.154360 (+ 1696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:01:41.154367 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:01:41.154563 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:41.154599 (+ 36us) client_negotiation.cc:769] Sending connection context
0807 23:01:41.154705 (+ 106us) client_negotiation.cc:240] Negotiation successful
0807 23:01:41.154849 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":846,"thread_start_us":643,"threads_started":1}
I20260807 23:01:41.155323 26116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93ca732f90794f3890e4ab071ee4d10d" candidate_uuid: "31051d460d394d6da8385c2f82b3ab7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cc8d20073804cc2b0e6f22f2ea059cd" is_pre_election: true
I20260807 23:01:41.156006 26116 raft_consensus.cc:2454] T 93ca732f90794f3890e4ab071ee4d10d P 5cc8d20073804cc2b0e6f22f2ea059cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31051d460d394d6da8385c2f82b3ab7e in term 0.
I20260807 23:01:41.156571 25898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93ca732f90794f3890e4ab071ee4d10d" candidate_uuid: "31051d460d394d6da8385c2f82b3ab7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "376a6ab84b28482094a9eb924d11e5f7" is_pre_election: true
I20260807 23:01:41.156746 26237 leader_election.cc:302] T 93ca732f90794f3890e4ab071ee4d10d P 31051d460d394d6da8385c2f82b3ab7e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31051d460d394d6da8385c2f82b3ab7e, 5cc8d20073804cc2b0e6f22f2ea059cd; no voters:
I20260807 23:01:41.157125 25898 raft_consensus.cc:2454] T 93ca732f90794f3890e4ab071ee4d10d P 376a6ab84b28482094a9eb924d11e5f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31051d460d394d6da8385c2f82b3ab7e in term 0.
I20260807 23:01:41.157241 26464 raft_consensus.cc:2781] T 93ca732f90794f3890e4ab071ee4d10d P 31051d460d394d6da8385c2f82b3ab7e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 23:01:41.157419 26464 raft_consensus.cc:482] T 93ca732f90794f3890e4ab071ee4d10d P 31051d460d394d6da8385c2f82b3ab7e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 23:01:41.157547 26464 raft_consensus.cc:3037] T 93ca732f90794f3890e4ab071ee4d10d P 31051d460d394d6da8385c2f82b3ab7e [term 0 FOLLOWER]: Advancing to term 1
I20260807 23:01:41.159971 26464 raft_consensus.cc:504] T 93ca732f90794f3890e4ab071ee4d10d P 31051d460d394d6da8385c2f82b3ab7e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31051d460d394d6da8385c2f82b3ab7e" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 46437 } } peers { permanent_uuid: "376a6ab84b28482094a9eb924d11e5f7" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 45303 } } peers { permanent_uuid: "5cc8d20073804cc2b0e6f22f2ea059cd" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 36545 } }
I20260807 23:01:41.160740 26464 leader_election.cc:288] T 93ca732f90794f3890e4ab071ee4d10d P 31051d460d394d6da8385c2f82b3ab7e [CANDIDATE]: Term 1 election: Requested vote from peers 376a6ab84b28482094a9eb924d11e5f7 (127.25.94.1:45303), 5cc8d20073804cc2b0e6f22f2ea059cd (127.25.94.2:36545)
I20260807 23:01:41.161427 26116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93ca732f90794f3890e4ab071ee4d10d" candidate_uuid: "31051d460d394d6da8385c2f82b3ab7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cc8d20073804cc2b0e6f22f2ea059cd"
I20260807 23:01:41.161662 26116 raft_consensus.cc:3037] T 93ca732f90794f3890e4ab071ee4d10d P 5cc8d20073804cc2b0e6f22f2ea059cd [term 0 FOLLOWER]: Advancing to term 1
I20260807 23:01:41.161382 25898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93ca732f90794f3890e4ab071ee4d10d" candidate_uuid: "31051d460d394d6da8385c2f82b3ab7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "376a6ab84b28482094a9eb924d11e5f7"
I20260807 23:01:41.161921 25898 raft_consensus.cc:3037] T 93ca732f90794f3890e4ab071ee4d10d P 376a6ab84b28482094a9eb924d11e5f7 [term 0 FOLLOWER]: Advancing to term 1
I20260807 23:01:41.164537 25898 raft_consensus.cc:2454] T 93ca732f90794f3890e4ab071ee4d10d P 376a6ab84b28482094a9eb924d11e5f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31051d460d394d6da8385c2f82b3ab7e in term 1.
I20260807 23:01:41.164587 26116 raft_consensus.cc:2454] T 93ca732f90794f3890e4ab071ee4d10d P 5cc8d20073804cc2b0e6f22f2ea059cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31051d460d394d6da8385c2f82b3ab7e in term 1.
I20260807 23:01:41.165102 26234 leader_election.cc:302] T 93ca732f90794f3890e4ab071ee4d10d P 31051d460d394d6da8385c2f82b3ab7e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31051d460d394d6da8385c2f82b3ab7e, 376a6ab84b28482094a9eb924d11e5f7; no voters:
I20260807 23:01:41.165555 26464 raft_consensus.cc:2781] T 93ca732f90794f3890e4ab071ee4d10d P 31051d460d394d6da8385c2f82b3ab7e [term 1 FOLLOWER]: Leader election won for term 1
I20260807 23:01:41.166112 26464 raft_consensus.cc:686] T 93ca732f90794f3890e4ab071ee4d10d P 31051d460d394d6da8385c2f82b3ab7e [term 1 LEADER]: Becoming Leader. State: Replica: 31051d460d394d6da8385c2f82b3ab7e, State: Running, Role: LEADER
I20260807 23:01:41.166697 26464 consensus_queue.cc:237] T 93ca732f90794f3890e4ab071ee4d10d P 31051d460d394d6da8385c2f82b3ab7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31051d460d394d6da8385c2f82b3ab7e" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 46437 } } peers { permanent_uuid: "376a6ab84b28482094a9eb924d11e5f7" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 45303 } } peers { permanent_uuid: "5cc8d20073804cc2b0e6f22f2ea059cd" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 36545 } }
I20260807 23:01:41.172514 25685 catalog_manager.cc:5495] T 93ca732f90794f3890e4ab071ee4d10d P 31051d460d394d6da8385c2f82b3ab7e reported cstate change: term changed from 0 to 1, leader changed from <none> to 31051d460d394d6da8385c2f82b3ab7e (127.25.94.3). New cstate: current_term: 1 leader_uuid: "31051d460d394d6da8385c2f82b3ab7e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31051d460d394d6da8385c2f82b3ab7e" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 46437 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "376a6ab84b28482094a9eb924d11e5f7" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 45303 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5cc8d20073804cc2b0e6f22f2ea059cd" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 36545 } health_report { overall_health: UNKNOWN } } }
I20260807 23:01:41.237293 26446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:41.231527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58236 (local address 127.25.94.3:46437)
0807 23:01:41.231704 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0807 23:01:41.231708 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 23:01:41.231806 (+ 98us) server_negotiation.cc:407] Connection header received
0807 23:01:41.231990 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:01:41.231993 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:01:41.232073 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:01:41.232213 (+ 140us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 23:01:41.232559 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:41.233373 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:41.234872 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:41.235275 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:41.235386 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:41.236011 (+ 625us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 23:01:41.236591 (+ 580us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 23:01:41.236781 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0807 23:01:41.236843 (+ 62us) server_negotiation.cc:299] Negotiation successful
0807 23:01:41.236929 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260807 23:01:41.245502 26116 raft_consensus.cc:1261] T 93ca732f90794f3890e4ab071ee4d10d P 5cc8d20073804cc2b0e6f22f2ea059cd [term 1 FOLLOWER]: Refusing update from remote peer 31051d460d394d6da8385c2f82b3ab7e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 23:01:41.245517 25898 raft_consensus.cc:1261] T 93ca732f90794f3890e4ab071ee4d10d P 376a6ab84b28482094a9eb924d11e5f7 [term 1 FOLLOWER]: Refusing update from remote peer 31051d460d394d6da8385c2f82b3ab7e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 23:01:41.246541 26468 consensus_queue.cc:1035] T 93ca732f90794f3890e4ab071ee4d10d P 31051d460d394d6da8385c2f82b3ab7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5cc8d20073804cc2b0e6f22f2ea059cd" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 36545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 23:01:41.246889 26464 consensus_queue.cc:1035] T 93ca732f90794f3890e4ab071ee4d10d P 31051d460d394d6da8385c2f82b3ab7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "376a6ab84b28482094a9eb924d11e5f7" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 45303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 23:01:41.275856 25684 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37048:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 23:01:41.276232 25684 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37048:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 23:01:41.280571 25684 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9e82911ca102413b95f33756345c831a: Sending DeleteTablet for 3 replicas of tablet 93ca732f90794f3890e4ab071ee4d10d
I20260807 23:01:41.281958 26083 tablet_service.cc:1514] Processing DeleteTablet for tablet 93ca732f90794f3890e4ab071ee4d10d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 23:01:41 UTC) from {username='kudu', principal='kudu/127.25.94.62@KRBTEST.COM'} at 127.0.0.1:46760
I20260807 23:01:41.282011 25863 tablet_service.cc:1514] Processing DeleteTablet for tablet 93ca732f90794f3890e4ab071ee4d10d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 23:01:41 UTC) from {username='kudu', principal='kudu/127.25.94.62@KRBTEST.COM'} at 127.0.0.1:39788
I20260807 23:01:41.282809 26313 tablet_service.cc:1514] Processing DeleteTablet for tablet 93ca732f90794f3890e4ab071ee4d10d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 23:01:41 UTC) from {username='kudu', principal='kudu/127.25.94.62@KRBTEST.COM'} at 127.0.0.1:58226
I20260807 23:01:41.283661 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 25777
I20260807 23:01:41.288542 26486 tablet_replica.cc:330] T 93ca732f90794f3890e4ab071ee4d10d P 5cc8d20073804cc2b0e6f22f2ea059cd: stopping tablet replica
I20260807 23:01:41.289129 26486 raft_consensus.cc:2229] T 93ca732f90794f3890e4ab071ee4d10d P 5cc8d20073804cc2b0e6f22f2ea059cd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 23:01:41.289789 26486 raft_consensus.cc:2258] T 93ca732f90794f3890e4ab071ee4d10d P 5cc8d20073804cc2b0e6f22f2ea059cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 23:01:41.290659 26488 tablet_replica.cc:330] T 93ca732f90794f3890e4ab071ee4d10d P 376a6ab84b28482094a9eb924d11e5f7: stopping tablet replica
I20260807 23:01:41.291056 26489 tablet_replica.cc:330] T 93ca732f90794f3890e4ab071ee4d10d P 31051d460d394d6da8385c2f82b3ab7e: stopping tablet replica
I20260807 23:01:41.291513 26489 raft_consensus.cc:2229] T 93ca732f90794f3890e4ab071ee4d10d P 31051d460d394d6da8385c2f82b3ab7e [term 1 LEADER]: Raft consensus shutting down.
I20260807 23:01:41.292152 26489 raft_consensus.cc:2258] T 93ca732f90794f3890e4ab071ee4d10d P 31051d460d394d6da8385c2f82b3ab7e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 23:01:41.292547 26486 ts_tablet_manager.cc:1905] T 93ca732f90794f3890e4ab071ee4d10d P 5cc8d20073804cc2b0e6f22f2ea059cd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 23:01:41.294322 26489 ts_tablet_manager.cc:1905] T 93ca732f90794f3890e4ab071ee4d10d P 31051d460d394d6da8385c2f82b3ab7e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 23:01:41.295019 26488 raft_consensus.cc:2229] T 93ca732f90794f3890e4ab071ee4d10d P 376a6ab84b28482094a9eb924d11e5f7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 23:01:41.295662 26488 raft_consensus.cc:2258] T 93ca732f90794f3890e4ab071ee4d10d P 376a6ab84b28482094a9eb924d11e5f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 23:01:41.301905 26486 ts_tablet_manager.cc:1918] T 93ca732f90794f3890e4ab071ee4d10d P 5cc8d20073804cc2b0e6f22f2ea059cd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 23:01:41.302160 26489 ts_tablet_manager.cc:1918] T 93ca732f90794f3890e4ab071ee4d10d P 31051d460d394d6da8385c2f82b3ab7e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 23:01:41.302362 26489 log.cc:1196] T 93ca732f90794f3890e4ab071ee4d10d P 31051d460d394d6da8385c2f82b3ab7e: Deleting WAL directory at /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786143361779964-25976-0/minicluster-data/ts-2/wal/wals/93ca732f90794f3890e4ab071ee4d10d
Aug 07 23:01:41 dist-test-slave-z8x9 krb5kdc[25610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143698, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.1@KRBTEST.COM
I20260807 23:01:41.302784 26486 log.cc:1196] T 93ca732f90794f3890e4ab071ee4d10d P 5cc8d20073804cc2b0e6f22f2ea059cd: Deleting WAL directory at /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786143361779964-25976-0/minicluster-data/ts-1/wal/wals/93ca732f90794f3890e4ab071ee4d10d
I20260807 23:01:41.303082 26489 ts_tablet_manager.cc:1939] T 93ca732f90794f3890e4ab071ee4d10d P 31051d460d394d6da8385c2f82b3ab7e: Deleting consensus metadata
I20260807 23:01:41.303453 26486 ts_tablet_manager.cc:1939] T 93ca732f90794f3890e4ab071ee4d10d P 5cc8d20073804cc2b0e6f22f2ea059cd: Deleting consensus metadata
I20260807 23:01:41.303961 26488 ts_tablet_manager.cc:1905] T 93ca732f90794f3890e4ab071ee4d10d P 376a6ab84b28482094a9eb924d11e5f7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 23:01:41.305101 25661 catalog_manager.cc:4841] TS 31051d460d394d6da8385c2f82b3ab7e (127.25.94.3:46437): tablet 93ca732f90794f3890e4ab071ee4d10d (table test-table [id=7e7e86e919c74415997d2d897339342b]) successfully deleted
I20260807 23:01:41.305321 25663 catalog_manager.cc:4841] TS 5cc8d20073804cc2b0e6f22f2ea059cd (127.25.94.2:36545): tablet 93ca732f90794f3890e4ab071ee4d10d (table test-table [id=7e7e86e919c74415997d2d897339342b]) successfully deleted
I20260807 23:01:41.309969 26488 ts_tablet_manager.cc:1918] T 93ca732f90794f3890e4ab071ee4d10d P 376a6ab84b28482094a9eb924d11e5f7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 23:01:41.310164 26488 log.cc:1196] T 93ca732f90794f3890e4ab071ee4d10d P 376a6ab84b28482094a9eb924d11e5f7: Deleting WAL directory at /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786143361779964-25976-0/minicluster-data/ts-0/wal/wals/93ca732f90794f3890e4ab071ee4d10d
I20260807 23:01:41.310762 26488 ts_tablet_manager.cc:1939] T 93ca732f90794f3890e4ab071ee4d10d P 376a6ab84b28482094a9eb924d11e5f7: Deleting consensus metadata
I20260807 23:01:41.312768 25661 catalog_manager.cc:4841] TS 376a6ab84b28482094a9eb924d11e5f7 (127.25.94.1:45303): tablet 93ca732f90794f3890e4ab071ee4d10d (table test-table [id=7e7e86e919c74415997d2d897339342b]) successfully deleted
I20260807 23:01:41.314693 26452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:41.285142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39794 (local address 127.25.94.1:45303)
0807 23:01:41.285382 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0807 23:01:41.285389 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 23:01:41.288166 (+ 2777us) server_negotiation.cc:407] Connection header received
0807 23:01:41.289420 (+ 1254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:01:41.289428 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:01:41.289516 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:01:41.289592 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:01:41.294047 (+ 4455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:41.295184 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:41.297193 (+ 2009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:41.297917 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:41.298027 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:41.306377 (+ 8350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:01:41.306395 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:01:41.306415 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 23:01:41.306470 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:01:41.311133 (+ 4663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:01:41.312263 (+ 1130us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:01:41.312274 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:01:41.312278 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:01:41.312476 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:01:41.313053 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:01:41.313058 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:01:41.313061 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:01:41.313524 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:01:41.313764 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0807 23:01:41.314161 (+ 397us) server_negotiation.cc:299] Negotiation successful
0807 23:01:41.314363 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260807 23:01:41.315510 25966 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 23:01:41.407378 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 25777
I20260807 23:01:41.429400 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 25987
Aug 07 23:01:41 dist-test-slave-z8x9 krb5kdc[25610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143698, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.2@KRBTEST.COM
I20260807 23:01:41.445895 26447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:41.430236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46770 (local address 127.25.94.2:36545)
0807 23:01:41.430414 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0807 23:01:41.430422 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 23:01:41.430513 (+ 91us) server_negotiation.cc:407] Connection header received
0807 23:01:41.430750 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:01:41.430757 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:01:41.430836 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:01:41.430994 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:01:41.432303 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:41.433026 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:41.434421 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:41.434846 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:41.434987 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:41.439858 (+ 4871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:01:41.439876 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:01:41.439893 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 23:01:41.439931 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:01:41.442927 (+ 2996us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:01:41.443688 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:01:41.443696 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:01:41.443698 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:01:41.443839 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:01:41.444233 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:01:41.444241 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:01:41.444245 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:01:41.444617 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:01:41.444956 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0807 23:01:41.445287 (+ 331us) server_negotiation.cc:299] Negotiation successful
0807 23:01:41.445519 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260807 23:01:41.446667 26189 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 23:01:41.536523 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 25987
I20260807 23:01:41.564519 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 26208
Aug 07 23:01:41 dist-test-slave-z8x9 krb5kdc[25610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143698, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.3@KRBTEST.COM
I20260807 23:01:41.583590 26446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:41.565466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58252 (local address 127.25.94.3:46437)
0807 23:01:41.565655 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0807 23:01:41.565659 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 23:01:41.565680 (+ 21us) server_negotiation.cc:407] Connection header received
0807 23:01:41.565945 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:01:41.565950 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:01:41.566017 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:01:41.566106 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:01:41.568093 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:41.568961 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:41.570927 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:41.571396 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:41.571492 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:41.577102 (+ 5610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:01:41.577115 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:01:41.577134 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 23:01:41.577356 (+ 222us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:01:41.580542 (+ 3186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:01:41.581315 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:01:41.581322 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:01:41.581325 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:01:41.581481 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:01:41.581891 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:01:41.581896 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:01:41.581898 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:01:41.582279 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:01:41.582673 (+ 394us) server_negotiation.cc:1035] Waiting for connection context
0807 23:01:41.582957 (+ 284us) server_negotiation.cc:299] Negotiation successful
0807 23:01:41.583195 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260807 23:01:41.584343 26416 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 23:01:41.708098 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 26208
I20260807 23:01:41.737375 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 25626
I20260807 23:01:41.738626 25727 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 23:01:41.853621 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 25626
2026-08-07T23:01:41Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (5110 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-07T23:01:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T23:01:41Z Disabled control of system clock
I20260807 23:01:41.951839 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786143361779964-25976-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786143361779964-25976-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786143361779964-25976-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.62:35511
--webserver_interface=127.25.94.62
--webserver_port=0
--builtin_ntp_servers=127.25.94.20:43517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.94.62:35511
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260807 23:01:42.313086 26502 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 23:01:42.313431 26502 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 23:01:42.313642 26502 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 23:01:42.320991 26502 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 23:01:42.321118 26502 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 23:01:42.321223 26502 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 23:01:42.321334 26502 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 23:01:42.321431 26502 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 23:01:42.330931 26502 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:43517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786143361779964-25976-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786143361779964-25976-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.94.62:35511
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.62:35511
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.94.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786143361779964-25976-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 23:01:42.332316 26502 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 23:01:42.334345 26502 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 23:01:42.347285 26511 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 23:01:42.347199 26510 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 23:01:42.347301 26513 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 23:01:42.348464 26502 server_base.cc:1034] running on GCE node
I20260807 23:01:42.349416 26502 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 23:01:42.350970 26502 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 23:01:42.352239 26502 hybrid_clock.cc:648] HybridClock initialized: now 1786143702352204 us; error 62 us; skew 500 ppm
I20260807 23:01:42.352653 26502 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 23:01:42.355109 26502 webserver.cc:460] Webserver started at http://127.25.94.62:45395/ using document root <none> and password file <none>
I20260807 23:01:42.355850 26502 fs_manager.cc:359] Metadata directory not provided
I20260807 23:01:42.355989 26502 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 23:01:42.356323 26502 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 23:01:42.359411 26502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786143361779964-25976-0/minicluster-data/master-0/data/instance:
uuid: "b9a26476b8024b01949f7f1297c67df9"
format_stamp: "Formatted at 2026-08-07 23:01:42 on dist-test-slave-z8x9"
server_key: "624d5c536b7ca64a6f3b4703dbd02ec7"
server_key_iv: "43849149085035f24bc50fbb192c095f"
server_key_version: "encryptionkey@0"
I20260807 23:01:42.360261 26502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786143361779964-25976-0/minicluster-data/master-0/wal/instance:
uuid: "b9a26476b8024b01949f7f1297c67df9"
format_stamp: "Formatted at 2026-08-07 23:01:42 on dist-test-slave-z8x9"
server_key: "624d5c536b7ca64a6f3b4703dbd02ec7"
server_key_iv: "43849149085035f24bc50fbb192c095f"
server_key_version: "encryptionkey@0"
I20260807 23:01:42.365764 26502 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260807 23:01:42.369479 26524 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 23:01:42.370921 26502 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 23:01:42.371141 26502 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786143361779964-25976-0/minicluster-data/master-0/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786143361779964-25976-0/minicluster-data/master-0/wal
uuid: "b9a26476b8024b01949f7f1297c67df9"
format_stamp: "Formatted at 2026-08-07 23:01:42 on dist-test-slave-z8x9"
server_key: "624d5c536b7ca64a6f3b4703dbd02ec7"
server_key_iv: "43849149085035f24bc50fbb192c095f"
server_key_version: "encryptionkey@0"
I20260807 23:01:42.371413 26502 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786143361779964-25976-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786143361779964-25976-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786143361779964-25976-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 23:01:42.391144 26502 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 23:01:42.392021 26502 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 23:01:42.392330 26502 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 23:01:42.411887 26502 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.62:35511
I20260807 23:01:42.411931 26606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.62:35511 every 8 connection(s)
I20260807 23:01:42.413360 26502 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
I20260807 23:01:42.414762 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 26502
I20260807 23:01:42.414947 25976 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786143361779964-25976-0/minicluster-data/master-0/wal/instance
I20260807 23:01:42.415474 25976 external_mini_cluster.cc:1428] Setting key 4867767941568c6045116d29f1fa04ed
I20260807 23:01:42.418320 26607 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 23:01:42.430033 26615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:42.417736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35402 (local address 127.25.94.62:35511)
0807 23:01:42.418799 (+ 1063us) server_negotiation.cc:206] Beginning negotiation
0807 23:01:42.418810 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 23:01:42.418852 (+ 42us) server_negotiation.cc:407] Connection header received
0807 23:01:42.420509 (+ 1657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:01:42.420531 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:01:42.421023 (+ 492us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:01:42.421480 (+ 457us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:01:42.422130 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:42.423554 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:42.425475 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:42.426148 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:42.426330 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:42.427615 (+ 1285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:01:42.427623 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:01:42.427641 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 23:01:42.428192 (+ 551us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:01:42.428284 (+ 92us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 23:01:42.428305 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:01:42.428404 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0807 23:01:42.428643 (+ 239us) server_negotiation.cc:299] Negotiation successful
0807 23:01:42.429007 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":651,"thread_start_us":424,"threads_started":1}
I20260807 23:01:42.431425 26607 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9a26476b8024b01949f7f1297c67df9: Bootstrap starting.
I20260807 23:01:42.434677 26607 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b9a26476b8024b01949f7f1297c67df9: Neither blocks nor log segments found. Creating new log.
I20260807 23:01:42.435668 26607 log.cc:824] T 00000000000000000000000000000000 P b9a26476b8024b01949f7f1297c67df9: Log is configured to *not* fsync() on all Append() calls
I20260807 23:01:42.438791 26607 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9a26476b8024b01949f7f1297c67df9: No bootstrap required, opened a new log
I20260807 23:01:42.446359 26607 raft_consensus.cc:348] T 00000000000000000000000000000000 P b9a26476b8024b01949f7f1297c67df9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9a26476b8024b01949f7f1297c67df9" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 35511 } }
I20260807 23:01:42.446753 26607 raft_consensus.cc:374] T 00000000000000000000000000000000 P b9a26476b8024b01949f7f1297c67df9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 23:01:42.446859 26607 raft_consensus.cc:729] T 00000000000000000000000000000000 P b9a26476b8024b01949f7f1297c67df9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9a26476b8024b01949f7f1297c67df9, State: Initialized, Role: FOLLOWER
I20260807 23:01:42.447499 26607 consensus_queue.cc:260] T 00000000000000000000000000000000 P b9a26476b8024b01949f7f1297c67df9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9a26476b8024b01949f7f1297c67df9" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 35511 } }
I20260807 23:01:42.447782 26607 raft_consensus.cc:388] T 00000000000000000000000000000000 P b9a26476b8024b01949f7f1297c67df9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 23:01:42.447928 26607 raft_consensus.cc:482] T 00000000000000000000000000000000 P b9a26476b8024b01949f7f1297c67df9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 23:01:42.448132 26607 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b9a26476b8024b01949f7f1297c67df9 [term 0 FOLLOWER]: Advancing to term 1
I20260807 23:01:42.450434 26607 raft_consensus.cc:504] T 00000000000000000000000000000000 P b9a26476b8024b01949f7f1297c67df9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9a26476b8024b01949f7f1297c67df9" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 35511 } }
I20260807 23:01:42.450980 26607 leader_election.cc:302] T 00000000000000000000000000000000 P b9a26476b8024b01949f7f1297c67df9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b9a26476b8024b01949f7f1297c67df9; no voters:
I20260807 23:01:42.451673 26607 leader_election.cc:288] T 00000000000000000000000000000000 P b9a26476b8024b01949f7f1297c67df9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 23:01:42.451864 26617 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b9a26476b8024b01949f7f1297c67df9 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 23:01:42.452625 26617 raft_consensus.cc:686] T 00000000000000000000000000000000 P b9a26476b8024b01949f7f1297c67df9 [term 1 LEADER]: Becoming Leader. State: Replica: b9a26476b8024b01949f7f1297c67df9, State: Running, Role: LEADER
I20260807 23:01:42.453240 26617 consensus_queue.cc:237] T 00000000000000000000000000000000 P b9a26476b8024b01949f7f1297c67df9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9a26476b8024b01949f7f1297c67df9" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 35511 } }
I20260807 23:01:42.453971 26607 sys_catalog.cc:564] T 00000000000000000000000000000000 P b9a26476b8024b01949f7f1297c67df9 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 23:01:42.456367 26618 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9a26476b8024b01949f7f1297c67df9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b9a26476b8024b01949f7f1297c67df9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9a26476b8024b01949f7f1297c67df9" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 35511 } } }
I20260807 23:01:42.456805 26618 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9a26476b8024b01949f7f1297c67df9 [sys.catalog]: This master's current role is: LEADER
I20260807 23:01:42.456315 26619 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9a26476b8024b01949f7f1297c67df9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b9a26476b8024b01949f7f1297c67df9. Latest consensus state: current_term: 1 leader_uuid: "b9a26476b8024b01949f7f1297c67df9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9a26476b8024b01949f7f1297c67df9" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 35511 } } }
I20260807 23:01:42.457193 26619 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9a26476b8024b01949f7f1297c67df9 [sys.catalog]: This master's current role is: LEADER
I20260807 23:01:42.459802 26635 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 23:01:42.465838 26635 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 23:01:42.475939 26635 catalog_manager.cc:1347] Generated new cluster ID: 7ba0e95a2e3442d594ca6f62117f876c
I20260807 23:01:42.476080 26635 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 23:01:42.502218 26635 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 23:01:42.503659 26635 catalog_manager.cc:1504] Loading token signing keys...
I20260807 23:01:42.514447 26635 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b9a26476b8024b01949f7f1297c67df9: Generated new TSK 0
I20260807 23:01:42.515499 26635 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 23:01:42.530438 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786143361779964-25976-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786143361779964-25976-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786143361779964-25976-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786143361779964-25976-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.1:0
--local_ip_for_outbound_sockets=127.25.94.1
--webserver_interface=127.25.94.1
--webserver_port=0
--tserver_master_addrs=127.25.94.62:35511
--builtin_ntp_servers=127.25.94.20:43517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260807 23:01:42.887472 26645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 23:01:42.887754 26645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 23:01:42.887933 26645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 23:01:42.895464 26645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 23:01:42.895566 26645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 23:01:42.895788 26645 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.1
I20260807 23:01:42.905072 26645 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:43517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786143361779964-25976-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786143361779964-25976-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786143361779964-25976-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.94.1
--webserver_port=0
--tserver_master_addrs=127.25.94.62:35511
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.1
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786143361779964-25976-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 23:01:42.906488 26645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 23:01:42.908320 26645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 23:01:42.921500 26656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 23:01:42.921526 26653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 23:01:42.921454 26654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 23:01:42.922423 26645 server_base.cc:1034] running on GCE node
I20260807 23:01:42.923056 26645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 23:01:42.924058 26645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 23:01:42.925274 26645 hybrid_clock.cc:648] HybridClock initialized: now 1786143702925236 us; error 53 us; skew 500 ppm
I20260807 23:01:42.925627 26645 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 23:01:42.927929 26645 webserver.cc:460] Webserver started at http://127.25.94.1:39873/ using document root <none> and password file <none>
I20260807 23:01:42.928690 26645 fs_manager.cc:359] Metadata directory not provided
I20260807 23:01:42.928836 26645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 23:01:42.929163 26645 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 23:01:42.932255 26645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786143361779964-25976-0/minicluster-data/ts-0/data/instance:
uuid: "01e8e20334af414bbab2f6edcd49017a"
format_stamp: "Formatted at 2026-08-07 23:01:42 on dist-test-slave-z8x9"
server_key: "2245d51f02139f7f60fe3fcb94f9c570"
server_key_iv: "6b06240c0d61abe87938deeabf740141"
server_key_version: "encryptionkey@0"
I20260807 23:01:42.933102 26645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786143361779964-25976-0/minicluster-data/ts-0/wal/instance:
uuid: "01e8e20334af414bbab2f6edcd49017a"
format_stamp: "Formatted at 2026-08-07 23:01:42 on dist-test-slave-z8x9"
server_key: "2245d51f02139f7f60fe3fcb94f9c570"
server_key_iv: "6b06240c0d61abe87938deeabf740141"
server_key_version: "encryptionkey@0"
I20260807 23:01:42.938642 26645 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260807 23:01:42.942420 26666 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 23:01:42.943893 26645 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20260807 23:01:42.944041 26645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786143361779964-25976-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786143361779964-25976-0/minicluster-data/ts-0/wal
uuid: "01e8e20334af414bbab2f6edcd49017a"
format_stamp: "Formatted at 2026-08-07 23:01:42 on dist-test-slave-z8x9"
server_key: "2245d51f02139f7f60fe3fcb94f9c570"
server_key_iv: "6b06240c0d61abe87938deeabf740141"
server_key_version: "encryptionkey@0"
I20260807 23:01:42.944228 26645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786143361779964-25976-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786143361779964-25976-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786143361779964-25976-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 23:01:42.961229 26645 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 23:01:42.962105 26645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 23:01:42.962360 26645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 23:01:42.963346 26645 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 23:01:42.964840 26645 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 23:01:42.964931 26645 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 23:01:42.965019 26645 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 23:01:42.965075 26645 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 23:01:42.997820 26645 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.1:36629
I20260807 23:01:42.997866 26857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.1:36629 every 8 connection(s)
I20260807 23:01:42.999140 26645 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786143361779964-25976-0/minicluster-data/ts-0/data/info.pb
I20260807 23:01:43.005152 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 26645
I20260807 23:01:43.005380 25976 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786143361779964-25976-0/minicluster-data/ts-0/wal/instance
I20260807 23:01:43.006009 25976 external_mini_cluster.cc:1428] Setting key 086fff352839b5554ad415e1bed3ef5a
I20260807 23:01:43.008801 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786143361779964-25976-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786143361779964-25976-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786143361779964-25976-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786143361779964-25976-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.2:0
--local_ip_for_outbound_sockets=127.25.94.2
--webserver_interface=127.25.94.2
--webserver_port=0
--tserver_master_addrs=127.25.94.62:35511
--builtin_ntp_servers=127.25.94.20:43517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260807 23:01:43.012734 26863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:43.001806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.1:58653 (local address 127.25.94.62:35511)
0807 23:01:43.002265 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0807 23:01:43.002270 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 23:01:43.003425 (+ 1155us) server_negotiation.cc:407] Connection header received
0807 23:01:43.005383 (+ 1958us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:01:43.005390 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:01:43.005503 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:01:43.005613 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:01:43.006926 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:43.007815 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:43.009490 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:43.010252 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:43.010399 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:43.011694 (+ 1295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:01:43.011704 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:01:43.011711 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 23:01:43.011736 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:01:43.011791 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 23:01:43.011811 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:01:43.011911 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0807 23:01:43.012197 (+ 286us) server_negotiation.cc:299] Negotiation successful
0807 23:01:43.012307 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":246,"threads_started":1}
I20260807 23:01:43.013650 26864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:43.002381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:35511 (local address 127.25.94.1:58653)
0807 23:01:43.003236 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0807 23:01:43.003301 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0807 23:01:43.005036 (+ 1735us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 23:01:43.005882 (+ 846us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 23:01:43.005894 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 23:01:43.006005 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0807 23:01:43.006763 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:01:43.006784 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:01:43.007960 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:01:43.007965 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:01:43.009348 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:01:43.009360 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:01:43.010467 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:01:43.010472 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:01:43.010661 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:43.011376 (+ 715us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 23:01:43.011379 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0807 23:01:43.011415 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 23:01:43.011488 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 23:01:43.011960 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 23:01:43.011969 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 23:01:43.011976 (+ 7us) client_negotiation.cc:769] Sending connection context
0807 23:01:43.012146 (+ 170us) client_negotiation.cc:240] Negotiation successful
0807 23:01:43.012455 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":283,"threads_started":1}
I20260807 23:01:43.015297 26858 heartbeater.cc:347] Connected to a master server at 127.25.94.62:35511
I20260807 23:01:43.015630 26858 heartbeater.cc:464] Registering TS with master...
I20260807 23:01:43.016595 26858 heartbeater.cc:511] Master 127.25.94.62:35511 requested a full tablet report, sending...
I20260807 23:01:43.019037 26549 ts_manager.cc:194] Registered new tserver with Master: 01e8e20334af414bbab2f6edcd49017a (127.25.94.1:36629)
I20260807 23:01:43.021140 26549 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.94.1:58653
W20260807 23:01:43.372262 26865 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 23:01:43.372565 26865 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 23:01:43.372653 26865 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 23:01:43.380079 26865 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 23:01:43.380187 26865 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 23:01:43.380297 26865 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.2
I20260807 23:01:43.389978 26865 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:43517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786143361779964-25976-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786143361779964-25976-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786143361779964-25976-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.94.2
--webserver_port=0
--tserver_master_addrs=127.25.94.62:35511
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.2
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786143361779964-25976-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 23:01:43.391326 26865 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 23:01:43.393391 26865 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 23:01:43.407155 26872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 23:01:43.407212 26876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 23:01:43.407665 26878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 23:01:43.408048 26865 server_base.cc:1034] running on GCE node
I20260807 23:01:43.408674 26865 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 23:01:43.409713 26865 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 23:01:43.410964 26865 hybrid_clock.cc:648] HybridClock initialized: now 1786143703410896 us; error 52 us; skew 500 ppm
I20260807 23:01:43.411331 26865 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 23:01:43.413864 26865 webserver.cc:460] Webserver started at http://127.25.94.2:42631/ using document root <none> and password file <none>
I20260807 23:01:43.414567 26865 fs_manager.cc:359] Metadata directory not provided
I20260807 23:01:43.414705 26865 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 23:01:43.415032 26865 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 23:01:43.417955 26865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786143361779964-25976-0/minicluster-data/ts-1/data/instance:
uuid: "34cb6cda69a14e70a533dbdb9c9c3542"
format_stamp: "Formatted at 2026-08-07 23:01:43 on dist-test-slave-z8x9"
server_key: "8fb7bb867dba063a6c6eae56c1515386"
server_key_iv: "1f0f515f9ac589bbf13c2a728371ef44"
server_key_version: "encryptionkey@0"
I20260807 23:01:43.418792 26865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786143361779964-25976-0/minicluster-data/ts-1/wal/instance:
uuid: "34cb6cda69a14e70a533dbdb9c9c3542"
format_stamp: "Formatted at 2026-08-07 23:01:43 on dist-test-slave-z8x9"
server_key: "8fb7bb867dba063a6c6eae56c1515386"
server_key_iv: "1f0f515f9ac589bbf13c2a728371ef44"
server_key_version: "encryptionkey@0"
I20260807 23:01:43.424064 26865 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260807 23:01:43.427542 26893 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 23:01:43.428848 26865 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260807 23:01:43.429040 26865 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786143361779964-25976-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786143361779964-25976-0/minicluster-data/ts-1/wal
uuid: "34cb6cda69a14e70a533dbdb9c9c3542"
format_stamp: "Formatted at 2026-08-07 23:01:43 on dist-test-slave-z8x9"
server_key: "8fb7bb867dba063a6c6eae56c1515386"
server_key_iv: "1f0f515f9ac589bbf13c2a728371ef44"
server_key_version: "encryptionkey@0"
I20260807 23:01:43.429316 26865 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786143361779964-25976-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786143361779964-25976-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786143361779964-25976-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 23:01:43.446743 26865 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 23:01:43.447633 26865 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 23:01:43.447932 26865 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 23:01:43.449023 26865 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 23:01:43.450645 26865 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 23:01:43.450791 26865 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 23:01:43.450932 26865 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 23:01:43.451020 26865 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 23:01:43.485452 26865 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.2:36589
I20260807 23:01:43.485505 27064 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.2:36589 every 8 connection(s)
I20260807 23:01:43.486766 26865 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786143361779964-25976-0/minicluster-data/ts-1/data/info.pb
I20260807 23:01:43.494477 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 26865
I20260807 23:01:43.494676 25976 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786143361779964-25976-0/minicluster-data/ts-1/wal/instance
I20260807 23:01:43.495091 25976 external_mini_cluster.cc:1428] Setting key a59d91ac57902c104644847ceb7b79ac
I20260807 23:01:43.498178 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786143361779964-25976-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786143361779964-25976-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786143361779964-25976-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786143361779964-25976-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.94.3:0
--local_ip_for_outbound_sockets=127.25.94.3
--webserver_interface=127.25.94.3
--webserver_port=0
--tserver_master_addrs=127.25.94.62:35511
--builtin_ntp_servers=127.25.94.20:43517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260807 23:01:43.500078 26863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:43.489510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.2:46703 (local address 127.25.94.62:35511)
0807 23:01:43.489727 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0807 23:01:43.489731 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 23:01:43.491060 (+ 1329us) server_negotiation.cc:407] Connection header received
0807 23:01:43.492921 (+ 1861us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:01:43.492928 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:01:43.493020 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:01:43.493134 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:01:43.494321 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:43.495189 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:43.497238 (+ 2049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:43.497636 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:43.497771 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:43.499252 (+ 1481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:01:43.499257 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:01:43.499261 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 23:01:43.499279 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:01:43.499313 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 23:01:43.499325 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:01:43.499420 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0807 23:01:43.499647 (+ 227us) server_negotiation.cc:299] Negotiation successful
0807 23:01:43.499765 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260807 23:01:43.501271 27071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:43.489909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:35511 (local address 127.25.94.2:46703)
0807 23:01:43.490866 (+ 957us) negotiation.cc:107] Waiting for socket to connect
0807 23:01:43.490906 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0807 23:01:43.492595 (+ 1689us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 23:01:43.493409 (+ 814us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 23:01:43.493417 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 23:01:43.493502 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0807 23:01:43.494113 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:01:43.494139 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:01:43.495381 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:01:43.495386 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:01:43.497035 (+ 1649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:01:43.497057 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:01:43.497883 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:01:43.497893 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:01:43.498122 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:43.498904 (+ 782us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 23:01:43.498907 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0807 23:01:43.498949 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 23:01:43.499033 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 23:01:43.499422 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 23:01:43.499436 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 23:01:43.499449 (+ 13us) client_negotiation.cc:769] Sending connection context
0807 23:01:43.499635 (+ 186us) client_negotiation.cc:240] Negotiation successful
0807 23:01:43.500057 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":322,"threads_started":1}
I20260807 23:01:43.502847 27068 heartbeater.cc:347] Connected to a master server at 127.25.94.62:35511
I20260807 23:01:43.503156 27068 heartbeater.cc:464] Registering TS with master...
I20260807 23:01:43.503954 27068 heartbeater.cc:511] Master 127.25.94.62:35511 requested a full tablet report, sending...
I20260807 23:01:43.505412 26549 ts_manager.cc:194] Registered new tserver with Master: 34cb6cda69a14e70a533dbdb9c9c3542 (127.25.94.2:36589)
I20260807 23:01:43.506201 26549 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.94.2:46703
W20260807 23:01:43.853428 27072 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 23:01:43.853803 27072 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 23:01:43.853966 27072 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 23:01:43.861608 27072 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 23:01:43.861713 27072 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 23:01:43.861897 27072 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.94.3
I20260807 23:01:43.871230 27072 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:43517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786143361779964-25976-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786143361779964-25976-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.94.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786143361779964-25976-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.94.3
--webserver_port=0
--tserver_master_addrs=127.25.94.62:35511
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.94.3
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786143361779964-25976-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 23:01:43.872607 27072 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 23:01:43.874461 27072 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 23:01:43.887771 27078 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 23:01:43.887827 27082 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 23:01:43.887851 27079 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 23:01:43.888681 27072 server_base.cc:1034] running on GCE node
I20260807 23:01:43.889452 27072 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 23:01:43.890441 27072 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 23:01:43.891748 27072 hybrid_clock.cc:648] HybridClock initialized: now 1786143703891710 us; error 45 us; skew 500 ppm
I20260807 23:01:43.892122 27072 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 23:01:43.910414 27072 webserver.cc:460] Webserver started at http://127.25.94.3:41537/ using document root <none> and password file <none>
I20260807 23:01:43.911254 27072 fs_manager.cc:359] Metadata directory not provided
I20260807 23:01:43.911362 27072 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 23:01:43.911775 27072 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 23:01:43.914886 27072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786143361779964-25976-0/minicluster-data/ts-2/data/instance:
uuid: "8245571e74574a0f91b392ac6dc72300"
format_stamp: "Formatted at 2026-08-07 23:01:43 on dist-test-slave-z8x9"
server_key: "5de9a86fa898b2f511533d3bf1cff067"
server_key_iv: "ecdac96a56d26789d827e3693668e853"
server_key_version: "encryptionkey@0"
I20260807 23:01:43.915735 27072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786143361779964-25976-0/minicluster-data/ts-2/wal/instance:
uuid: "8245571e74574a0f91b392ac6dc72300"
format_stamp: "Formatted at 2026-08-07 23:01:43 on dist-test-slave-z8x9"
server_key: "5de9a86fa898b2f511533d3bf1cff067"
server_key_iv: "ecdac96a56d26789d827e3693668e853"
server_key_version: "encryptionkey@0"
I20260807 23:01:43.921413 27072 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260807 23:01:43.925139 27088 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 23:01:43.926618 27072 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 23:01:43.926829 27072 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786143361779964-25976-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786143361779964-25976-0/minicluster-data/ts-2/wal
uuid: "8245571e74574a0f91b392ac6dc72300"
format_stamp: "Formatted at 2026-08-07 23:01:43 on dist-test-slave-z8x9"
server_key: "5de9a86fa898b2f511533d3bf1cff067"
server_key_iv: "ecdac96a56d26789d827e3693668e853"
server_key_version: "encryptionkey@0"
I20260807 23:01:43.927035 27072 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786143361779964-25976-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786143361779964-25976-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786143361779964-25976-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 23:01:43.947352 27072 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 23:01:43.948549 27072 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 23:01:43.948853 27072 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 23:01:43.950057 27072 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 23:01:43.951745 27072 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 23:01:43.951846 27072 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 23:01:43.952013 27072 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 23:01:43.952081 27072 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 23:01:43.986918 27072 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.3:44197
I20260807 23:01:43.987016 27259 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.3:44197 every 8 connection(s)
I20260807 23:01:43.988265 27072 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786143361779964-25976-0/minicluster-data/ts-2/data/info.pb
I20260807 23:01:43.992174 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 27072
I20260807 23:01:43.992374 25976 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786143361779964-25976-0/minicluster-data/ts-2/wal/instance
I20260807 23:01:43.992930 25976 external_mini_cluster.cc:1428] Setting key 77c3824582b298df3b791711dbe5da4d
I20260807 23:01:44.004910 26863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:43.990998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.3:53585 (local address 127.25.94.62:35511)
0807 23:01:43.991225 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0807 23:01:43.991232 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 23:01:43.992876 (+ 1644us) server_negotiation.cc:407] Connection header received
0807 23:01:43.995425 (+ 2549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:01:43.995432 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:01:43.995525 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:01:43.995670 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:01:43.998562 (+ 2892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:43.999349 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:44.001818 (+ 2469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:44.002353 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:44.002457 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:44.004078 (+ 1621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:01:44.004084 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:01:44.004087 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 23:01:44.004105 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:01:44.004137 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 23:01:44.004149 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:01:44.004222 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0807 23:01:44.004509 (+ 287us) server_negotiation.cc:299] Negotiation successful
0807 23:01:44.004597 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260807 23:01:44.005910 27265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:43.991452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:35511 (local address 127.25.94.3:53585)
0807 23:01:43.992667 (+ 1215us) negotiation.cc:107] Waiting for socket to connect
0807 23:01:43.992724 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0807 23:01:43.995127 (+ 2403us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 23:01:43.996599 (+ 1472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 23:01:43.996618 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 23:01:43.996818 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0807 23:01:43.998265 (+ 1447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:01:43.998337 (+ 72us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:01:43.999533 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:01:43.999538 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:01:44.001637 (+ 2099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:01:44.001657 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:01:44.002590 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:01:44.002597 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:01:44.002885 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:44.003702 (+ 817us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 23:01:44.003705 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0807 23:01:44.003758 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 23:01:44.003867 (+ 109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 23:01:44.004257 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 23:01:44.004268 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 23:01:44.004285 (+ 17us) client_negotiation.cc:769] Sending connection context
0807 23:01:44.004484 (+ 199us) client_negotiation.cc:240] Negotiation successful
0807 23:01:44.004793 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":602,"thread_start_us":363,"threads_started":1}
I20260807 23:01:44.007579 27260 heartbeater.cc:347] Connected to a master server at 127.25.94.62:35511
I20260807 23:01:44.007967 27260 heartbeater.cc:464] Registering TS with master...
I20260807 23:01:44.008921 27260 heartbeater.cc:511] Master 127.25.94.62:35511 requested a full tablet report, sending...
I20260807 23:01:44.010372 26549 ts_manager.cc:194] Registered new tserver with Master: 8245571e74574a0f91b392ac6dc72300 (127.25.94.3:44197)
I20260807 23:01:44.011132 26549 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.94.3:53585
I20260807 23:01:44.021739 25976 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 23:01:44.024785 26858 heartbeater.cc:503] Master 127.25.94.62:35511 was elected leader, sending a full tablet report...
I20260807 23:01:44.032183 26863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:44.026137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35414 (local address 127.25.94.62:35511)
0807 23:01:44.026421 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0807 23:01:44.026431 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 23:01:44.026856 (+ 425us) server_negotiation.cc:407] Connection header received
0807 23:01:44.027033 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:01:44.027037 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:01:44.027106 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:01:44.027188 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:01:44.027633 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:44.028347 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:44.029754 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:44.030161 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:44.030318 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:44.031448 (+ 1130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:01:44.031453 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:01:44.031457 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 23:01:44.031474 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:01:44.031504 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 23:01:44.031516 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:01:44.031614 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0807 23:01:44.031761 (+ 147us) server_negotiation.cc:299] Negotiation successful
0807 23:01:44.031860 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260807 23:01:44.036051 26549 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35414:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 23:01:44.042973 26549 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 23:01:44.065897 27282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:44.056549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.3:44197 (local address 127.0.0.1:44788)
0807 23:01:44.057226 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0807 23:01:44.057249 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0807 23:01:44.057364 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 23:01:44.058664 (+ 1300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 23:01:44.058671 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 23:01:44.058711 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 23:01:44.059259 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:01:44.059280 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:01:44.060869 (+ 1589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:01:44.060875 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:01:44.063188 (+ 2313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:01:44.063207 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:01:44.065048 (+ 1841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:01:44.065057 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:01:44.065272 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:44.065308 (+ 36us) client_negotiation.cc:769] Sending connection context
0807 23:01:44.065398 (+ 90us) client_negotiation.cc:240] Negotiation successful
0807 23:01:44.065535 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":356,"threads_started":1}
I20260807 23:01:44.066215 27283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:44.057129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44788 (local address 127.25.94.3:44197)
0807 23:01:44.058019 (+ 890us) server_negotiation.cc:206] Beginning negotiation
0807 23:01:44.058029 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 23:01:44.058063 (+ 34us) server_negotiation.cc:407] Connection header received
0807 23:01:44.058158 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:01:44.058165 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:01:44.058450 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:01:44.058659 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 23:01:44.059491 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:44.060725 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:44.063409 (+ 2684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:44.064836 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:44.064987 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:44.065269 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0807 23:01:44.065661 (+ 392us) server_negotiation.cc:299] Negotiation successful
0807 23:01:44.065825 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":716,"thread_start_us":491,"threads_started":1}
I20260807 23:01:44.067065 27281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:44.055622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51866 (local address 127.25.94.2:36589)
0807 23:01:44.056225 (+ 603us) server_negotiation.cc:206] Beginning negotiation
0807 23:01:44.056239 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0807 23:01:44.056290 (+ 51us) server_negotiation.cc:407] Connection header received
0807 23:01:44.056604 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:01:44.056614 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:01:44.056899 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:01:44.057124 (+ 225us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 23:01:44.058168 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:44.059890 (+ 1722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:44.062772 (+ 2882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:44.064300 (+ 1528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:44.064436 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:44.064690 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0807 23:01:44.065389 (+ 699us) server_negotiation.cc:299] Negotiation successful
0807 23:01:44.065546 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":469,"thread_start_us":350,"threads_started":1}
I20260807 23:01:44.067553 27286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:44.058121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.1:36629 (local address 127.0.0.1:55538)
0807 23:01:44.059327 (+ 1206us) negotiation.cc:107] Waiting for socket to connect
0807 23:01:44.059345 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 23:01:44.059562 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 23:01:44.060399 (+ 837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 23:01:44.060404 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 23:01:44.060433 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 23:01:44.061050 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:01:44.061064 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:01:44.062988 (+ 1924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:01:44.062995 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:01:44.064421 (+ 1426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:01:44.064433 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:01:44.066719 (+ 2286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:01:44.066726 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:01:44.066918 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:44.066948 (+ 30us) client_negotiation.cc:769] Sending connection context
0807 23:01:44.067053 (+ 105us) client_negotiation.cc:240] Negotiation successful
0807 23:01:44.067188 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1075,"mutex_wait_us":101,"thread_start_us":910,"threads_started":1}
I20260807 23:01:44.067817 27280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:44.055448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.2:36589 (local address 127.0.0.1:51866)
0807 23:01:44.056170 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0807 23:01:44.056213 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0807 23:01:44.056363 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 23:01:44.057504 (+ 1141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 23:01:44.057511 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 23:01:44.057565 (+ 54us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 23:01:44.057940 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:01:44.057953 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:01:44.060060 (+ 2107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:01:44.060068 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:01:44.062531 (+ 2463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:01:44.062543 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:01:44.064822 (+ 2279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:01:44.064829 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:01:44.065050 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:44.065103 (+ 53us) client_negotiation.cc:769] Sending connection context
0807 23:01:44.065273 (+ 170us) client_negotiation.cc:240] Negotiation successful
0807 23:01:44.065472 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"mutex_wait_us":110,"thread_start_us":314,"threads_started":1}
I20260807 23:01:44.068063 27284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:44.057769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55538 (local address 127.25.94.1:36629)
0807 23:01:44.058352 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0807 23:01:44.058360 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 23:01:44.059465 (+ 1105us) server_negotiation.cc:407] Connection header received
0807 23:01:44.059726 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:01:44.059733 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:01:44.059949 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:01:44.060126 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 23:01:44.061279 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:44.062811 (+ 1532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:44.064614 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:44.066008 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:44.066435 (+ 427us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:44.066719 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0807 23:01:44.067111 (+ 392us) server_negotiation.cc:299] Negotiation successful
0807 23:01:44.067411 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":466,"threads_started":1}
I20260807 23:01:44.070544 27160 tablet_service.cc:1467] Processing CreateTablet for tablet 1f80ea6e3659494f9a613061a6641890 (DEFAULT_TABLE table=test-table [id=a42fac07ca78480ca6a1cb548edf8b65]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 23:01:44.071245 26962 tablet_service.cc:1467] Processing CreateTablet for tablet 1f80ea6e3659494f9a613061a6641890 (DEFAULT_TABLE table=test-table [id=a42fac07ca78480ca6a1cb548edf8b65]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 23:01:44.071261 26748 tablet_service.cc:1467] Processing CreateTablet for tablet 1f80ea6e3659494f9a613061a6641890 (DEFAULT_TABLE table=test-table [id=a42fac07ca78480ca6a1cb548edf8b65]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 23:01:44.072176 27160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f80ea6e3659494f9a613061a6641890. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 23:01:44.072501 26962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f80ea6e3659494f9a613061a6641890. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 23:01:44.072700 26748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f80ea6e3659494f9a613061a6641890. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 23:01:44.084586 27290 tablet_bootstrap.cc:492] T 1f80ea6e3659494f9a613061a6641890 P 01e8e20334af414bbab2f6edcd49017a: Bootstrap starting.
I20260807 23:01:44.087371 27293 tablet_bootstrap.cc:492] T 1f80ea6e3659494f9a613061a6641890 P 8245571e74574a0f91b392ac6dc72300: Bootstrap starting.
I20260807 23:01:44.089010 27295 tablet_bootstrap.cc:492] T 1f80ea6e3659494f9a613061a6641890 P 34cb6cda69a14e70a533dbdb9c9c3542: Bootstrap starting.
I20260807 23:01:44.089083 27290 tablet_bootstrap.cc:654] T 1f80ea6e3659494f9a613061a6641890 P 01e8e20334af414bbab2f6edcd49017a: Neither blocks nor log segments found. Creating new log.
I20260807 23:01:44.091087 27293 tablet_bootstrap.cc:654] T 1f80ea6e3659494f9a613061a6641890 P 8245571e74574a0f91b392ac6dc72300: Neither blocks nor log segments found. Creating new log.
I20260807 23:01:44.091100 27290 log.cc:824] T 1f80ea6e3659494f9a613061a6641890 P 01e8e20334af414bbab2f6edcd49017a: Log is configured to *not* fsync() on all Append() calls
I20260807 23:01:44.092255 27293 log.cc:824] T 1f80ea6e3659494f9a613061a6641890 P 8245571e74574a0f91b392ac6dc72300: Log is configured to *not* fsync() on all Append() calls
I20260807 23:01:44.092476 27295 tablet_bootstrap.cc:654] T 1f80ea6e3659494f9a613061a6641890 P 34cb6cda69a14e70a533dbdb9c9c3542: Neither blocks nor log segments found. Creating new log.
I20260807 23:01:44.093746 27295 log.cc:824] T 1f80ea6e3659494f9a613061a6641890 P 34cb6cda69a14e70a533dbdb9c9c3542: Log is configured to *not* fsync() on all Append() calls
I20260807 23:01:44.094551 27290 tablet_bootstrap.cc:492] T 1f80ea6e3659494f9a613061a6641890 P 01e8e20334af414bbab2f6edcd49017a: No bootstrap required, opened a new log
I20260807 23:01:44.094849 27290 ts_tablet_manager.cc:1397] T 1f80ea6e3659494f9a613061a6641890 P 01e8e20334af414bbab2f6edcd49017a: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.007s
I20260807 23:01:44.095417 27293 tablet_bootstrap.cc:492] T 1f80ea6e3659494f9a613061a6641890 P 8245571e74574a0f91b392ac6dc72300: No bootstrap required, opened a new log
I20260807 23:01:44.095678 27293 ts_tablet_manager.cc:1397] T 1f80ea6e3659494f9a613061a6641890 P 8245571e74574a0f91b392ac6dc72300: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20260807 23:01:44.096428 27295 tablet_bootstrap.cc:492] T 1f80ea6e3659494f9a613061a6641890 P 34cb6cda69a14e70a533dbdb9c9c3542: No bootstrap required, opened a new log
I20260807 23:01:44.096690 27295 ts_tablet_manager.cc:1397] T 1f80ea6e3659494f9a613061a6641890 P 34cb6cda69a14e70a533dbdb9c9c3542: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260807 23:01:44.104282 27295 raft_consensus.cc:348] T 1f80ea6e3659494f9a613061a6641890 P 34cb6cda69a14e70a533dbdb9c9c3542 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34cb6cda69a14e70a533dbdb9c9c3542" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 36589 } } peers { permanent_uuid: "8245571e74574a0f91b392ac6dc72300" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 44197 } } peers { permanent_uuid: "01e8e20334af414bbab2f6edcd49017a" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 36629 } }
I20260807 23:01:44.104722 27295 raft_consensus.cc:374] T 1f80ea6e3659494f9a613061a6641890 P 34cb6cda69a14e70a533dbdb9c9c3542 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 23:01:44.104832 27295 raft_consensus.cc:729] T 1f80ea6e3659494f9a613061a6641890 P 34cb6cda69a14e70a533dbdb9c9c3542 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34cb6cda69a14e70a533dbdb9c9c3542, State: Initialized, Role: FOLLOWER
I20260807 23:01:44.105559 27295 consensus_queue.cc:260] T 1f80ea6e3659494f9a613061a6641890 P 34cb6cda69a14e70a533dbdb9c9c3542 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34cb6cda69a14e70a533dbdb9c9c3542" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 36589 } } peers { permanent_uuid: "8245571e74574a0f91b392ac6dc72300" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 44197 } } peers { permanent_uuid: "01e8e20334af414bbab2f6edcd49017a" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 36629 } }
I20260807 23:01:44.105847 27290 raft_consensus.cc:348] T 1f80ea6e3659494f9a613061a6641890 P 01e8e20334af414bbab2f6edcd49017a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34cb6cda69a14e70a533dbdb9c9c3542" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 36589 } } peers { permanent_uuid: "8245571e74574a0f91b392ac6dc72300" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 44197 } } peers { permanent_uuid: "01e8e20334af414bbab2f6edcd49017a" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 36629 } }
I20260807 23:01:44.106355 27290 raft_consensus.cc:374] T 1f80ea6e3659494f9a613061a6641890 P 01e8e20334af414bbab2f6edcd49017a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 23:01:44.106478 27290 raft_consensus.cc:729] T 1f80ea6e3659494f9a613061a6641890 P 01e8e20334af414bbab2f6edcd49017a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01e8e20334af414bbab2f6edcd49017a, State: Initialized, Role: FOLLOWER
I20260807 23:01:44.106910 27068 heartbeater.cc:503] Master 127.25.94.62:35511 was elected leader, sending a full tablet report...
I20260807 23:01:44.106511 27293 raft_consensus.cc:348] T 1f80ea6e3659494f9a613061a6641890 P 8245571e74574a0f91b392ac6dc72300 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34cb6cda69a14e70a533dbdb9c9c3542" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 36589 } } peers { permanent_uuid: "8245571e74574a0f91b392ac6dc72300" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 44197 } } peers { permanent_uuid: "01e8e20334af414bbab2f6edcd49017a" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 36629 } }
I20260807 23:01:44.107302 27293 raft_consensus.cc:374] T 1f80ea6e3659494f9a613061a6641890 P 8245571e74574a0f91b392ac6dc72300 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 23:01:44.107412 27293 raft_consensus.cc:729] T 1f80ea6e3659494f9a613061a6641890 P 8245571e74574a0f91b392ac6dc72300 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8245571e74574a0f91b392ac6dc72300, State: Initialized, Role: FOLLOWER
I20260807 23:01:44.107347 27290 consensus_queue.cc:260] T 1f80ea6e3659494f9a613061a6641890 P 01e8e20334af414bbab2f6edcd49017a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34cb6cda69a14e70a533dbdb9c9c3542" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 36589 } } peers { permanent_uuid: "8245571e74574a0f91b392ac6dc72300" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 44197 } } peers { permanent_uuid: "01e8e20334af414bbab2f6edcd49017a" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 36629 } }
I20260807 23:01:44.107535 27295 ts_tablet_manager.cc:1428] T 1f80ea6e3659494f9a613061a6641890 P 34cb6cda69a14e70a533dbdb9c9c3542: Time spent starting tablet: real 0.011s user 0.008s sys 0.003s
I20260807 23:01:44.108062 27293 consensus_queue.cc:260] T 1f80ea6e3659494f9a613061a6641890 P 8245571e74574a0f91b392ac6dc72300 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34cb6cda69a14e70a533dbdb9c9c3542" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 36589 } } peers { permanent_uuid: "8245571e74574a0f91b392ac6dc72300" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 44197 } } peers { permanent_uuid: "01e8e20334af414bbab2f6edcd49017a" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 36629 } }
I20260807 23:01:44.110740 27290 ts_tablet_manager.cc:1428] T 1f80ea6e3659494f9a613061a6641890 P 01e8e20334af414bbab2f6edcd49017a: Time spent starting tablet: real 0.016s user 0.016s sys 0.000s
I20260807 23:01:44.110901 27260 heartbeater.cc:503] Master 127.25.94.62:35511 was elected leader, sending a full tablet report...
I20260807 23:01:44.111626 27293 ts_tablet_manager.cc:1428] T 1f80ea6e3659494f9a613061a6641890 P 8245571e74574a0f91b392ac6dc72300: Time spent starting tablet: real 0.016s user 0.014s sys 0.002s
W20260807 23:01:44.239061 27070 tablet.cc:2369] T 1f80ea6e3659494f9a613061a6641890 P 34cb6cda69a14e70a533dbdb9c9c3542: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 23:01:44.239980 27263 tablet.cc:2369] T 1f80ea6e3659494f9a613061a6641890 P 8245571e74574a0f91b392ac6dc72300: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 23:01:44.251734 26862 tablet.cc:2369] T 1f80ea6e3659494f9a613061a6641890 P 01e8e20334af414bbab2f6edcd49017a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 23:01:44.373751 27301 raft_consensus.cc:482] T 1f80ea6e3659494f9a613061a6641890 P 34cb6cda69a14e70a533dbdb9c9c3542 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 23:01:44.374076 27301 raft_consensus.cc:504] T 1f80ea6e3659494f9a613061a6641890 P 34cb6cda69a14e70a533dbdb9c9c3542 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34cb6cda69a14e70a533dbdb9c9c3542" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 36589 } } peers { permanent_uuid: "8245571e74574a0f91b392ac6dc72300" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 44197 } } peers { permanent_uuid: "01e8e20334af414bbab2f6edcd49017a" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 36629 } }
I20260807 23:01:44.375922 27301 leader_election.cc:288] T 1f80ea6e3659494f9a613061a6641890 P 34cb6cda69a14e70a533dbdb9c9c3542 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8245571e74574a0f91b392ac6dc72300 (127.25.94.3:44197), 01e8e20334af414bbab2f6edcd49017a (127.25.94.1:36629)
I20260807 23:01:44.381362 27303 raft_consensus.cc:482] T 1f80ea6e3659494f9a613061a6641890 P 01e8e20334af414bbab2f6edcd49017a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 23:01:44.381711 27303 raft_consensus.cc:504] T 1f80ea6e3659494f9a613061a6641890 P 01e8e20334af414bbab2f6edcd49017a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34cb6cda69a14e70a533dbdb9c9c3542" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 36589 } } peers { permanent_uuid: "8245571e74574a0f91b392ac6dc72300" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 44197 } } peers { permanent_uuid: "01e8e20334af414bbab2f6edcd49017a" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 36629 } }
I20260807 23:01:44.383693 27303 leader_election.cc:288] T 1f80ea6e3659494f9a613061a6641890 P 01e8e20334af414bbab2f6edcd49017a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 34cb6cda69a14e70a533dbdb9c9c3542 (127.25.94.2:36589), 8245571e74574a0f91b392ac6dc72300 (127.25.94.3:44197)
I20260807 23:01:44.384076 27305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:44.376309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.3:44197 (local address 127.25.94.2:37215)
0807 23:01:44.377129 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0807 23:01:44.377162 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0807 23:01:44.377402 (+ 240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 23:01:44.377836 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 23:01:44.377844 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 23:01:44.377878 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 23:01:44.378302 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:01:44.378317 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:01:44.379606 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:01:44.379622 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:01:44.381481 (+ 1859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:01:44.381494 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:01:44.383177 (+ 1683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:01:44.383197 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:01:44.383385 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:44.383420 (+ 35us) client_negotiation.cc:769] Sending connection context
0807 23:01:44.383524 (+ 104us) client_negotiation.cc:240] Negotiation successful
0807 23:01:44.383669 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":662,"thread_start_us":417,"threads_started":1}
I20260807 23:01:44.384095 27283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:44.376518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.2:37215 (local address 127.25.94.3:44197)
0807 23:01:44.376823 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0807 23:01:44.376833 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 23:01:44.377336 (+ 503us) server_negotiation.cc:407] Connection header received
0807 23:01:44.377569 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:01:44.377576 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:01:44.377701 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:01:44.377803 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 23:01:44.378476 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:44.379441 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:44.381777 (+ 2336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:44.382961 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:44.383099 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:44.383206 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0807 23:01:44.383676 (+ 470us) server_negotiation.cc:299] Negotiation successful
0807 23:01:44.383833 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260807 23:01:44.384850 27284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:44.377505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.2:47087 (local address 127.25.94.1:36629)
0807 23:01:44.377730 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0807 23:01:44.377737 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 23:01:44.378136 (+ 399us) server_negotiation.cc:407] Connection header received
0807 23:01:44.378244 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:01:44.378250 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:01:44.378332 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:01:44.378428 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 23:01:44.379082 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:44.380140 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:44.382240 (+ 2100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:44.383460 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:44.383757 (+ 297us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:44.383858 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0807 23:01:44.384468 (+ 610us) server_negotiation.cc:299] Negotiation successful
0807 23:01:44.384602 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260807 23:01:44.385462 27183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f80ea6e3659494f9a613061a6641890" candidate_uuid: "34cb6cda69a14e70a533dbdb9c9c3542" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8245571e74574a0f91b392ac6dc72300" is_pre_election: true
I20260807 23:01:44.385780 27306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:44.377116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.1:36629 (local address 127.25.94.2:47087)
0807 23:01:44.378009 (+ 893us) negotiation.cc:107] Waiting for socket to connect
0807 23:01:44.378034 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0807 23:01:44.378135 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 23:01:44.378464 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 23:01:44.378468 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 23:01:44.378492 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 23:01:44.378861 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:01:44.378872 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:01:44.380287 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:01:44.380291 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:01:44.382039 (+ 1748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:01:44.382054 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:01:44.383998 (+ 1944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:01:44.384004 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:01:44.384216 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:44.384292 (+ 76us) client_negotiation.cc:769] Sending connection context
0807 23:01:44.385284 (+ 992us) client_negotiation.cc:240] Negotiation successful
0807 23:01:44.385445 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":752,"thread_start_us":467,"threads_started":1}
I20260807 23:01:44.386144 27183 raft_consensus.cc:2454] T 1f80ea6e3659494f9a613061a6641890 P 8245571e74574a0f91b392ac6dc72300 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 34cb6cda69a14e70a533dbdb9c9c3542 in term 0.
I20260807 23:01:44.386770 26780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f80ea6e3659494f9a613061a6641890" candidate_uuid: "34cb6cda69a14e70a533dbdb9c9c3542" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01e8e20334af414bbab2f6edcd49017a" is_pre_election: true
I20260807 23:01:44.387071 26896 leader_election.cc:302] T 1f80ea6e3659494f9a613061a6641890 P 34cb6cda69a14e70a533dbdb9c9c3542 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34cb6cda69a14e70a533dbdb9c9c3542, 8245571e74574a0f91b392ac6dc72300; no voters:
I20260807 23:01:44.387239 26780 raft_consensus.cc:2454] T 1f80ea6e3659494f9a613061a6641890 P 01e8e20334af414bbab2f6edcd49017a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 34cb6cda69a14e70a533dbdb9c9c3542 in term 0.
I20260807 23:01:44.387637 27301 raft_consensus.cc:2781] T 1f80ea6e3659494f9a613061a6641890 P 34cb6cda69a14e70a533dbdb9c9c3542 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 23:01:44.387825 27301 raft_consensus.cc:482] T 1f80ea6e3659494f9a613061a6641890 P 34cb6cda69a14e70a533dbdb9c9c3542 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 23:01:44.387951 27301 raft_consensus.cc:3037] T 1f80ea6e3659494f9a613061a6641890 P 34cb6cda69a14e70a533dbdb9c9c3542 [term 0 FOLLOWER]: Advancing to term 1
I20260807 23:01:44.391050 27301 raft_consensus.cc:504] T 1f80ea6e3659494f9a613061a6641890 P 34cb6cda69a14e70a533dbdb9c9c3542 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34cb6cda69a14e70a533dbdb9c9c3542" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 36589 } } peers { permanent_uuid: "8245571e74574a0f91b392ac6dc72300" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 44197 } } peers { permanent_uuid: "01e8e20334af414bbab2f6edcd49017a" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 36629 } }
I20260807 23:01:44.392266 27183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f80ea6e3659494f9a613061a6641890" candidate_uuid: "34cb6cda69a14e70a533dbdb9c9c3542" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8245571e74574a0f91b392ac6dc72300"
I20260807 23:01:44.392549 27183 raft_consensus.cc:3037] T 1f80ea6e3659494f9a613061a6641890 P 8245571e74574a0f91b392ac6dc72300 [term 0 FOLLOWER]: Advancing to term 1
I20260807 23:01:44.392603 27281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:44.384577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.1:59675 (local address 127.25.94.2:36589)
0807 23:01:44.384834 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0807 23:01:44.384843 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 23:01:44.385404 (+ 561us) server_negotiation.cc:407] Connection header received
0807 23:01:44.385676 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:01:44.385681 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:01:44.385774 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:01:44.385843 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 23:01:44.386756 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:44.387827 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:44.390622 (+ 2795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:44.391600 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:44.391752 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:44.391849 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0807 23:01:44.392221 (+ 372us) server_negotiation.cc:299] Negotiation successful
0807 23:01:44.392344 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260807 23:01:44.392774 26780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f80ea6e3659494f9a613061a6641890" candidate_uuid: "34cb6cda69a14e70a533dbdb9c9c3542" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01e8e20334af414bbab2f6edcd49017a"
I20260807 23:01:44.392983 26780 raft_consensus.cc:3037] T 1f80ea6e3659494f9a613061a6641890 P 01e8e20334af414bbab2f6edcd49017a [term 0 FOLLOWER]: Advancing to term 1
I20260807 23:01:44.393441 27307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:44.384209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.2:36589 (local address 127.25.94.1:59675)
0807 23:01:44.385312 (+ 1103us) negotiation.cc:107] Waiting for socket to connect
0807 23:01:44.385330 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 23:01:44.385509 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 23:01:44.386073 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 23:01:44.386080 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 23:01:44.386141 (+ 61us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 23:01:44.386565 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:01:44.386578 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:01:44.388001 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:01:44.388006 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:01:44.390458 (+ 2452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:01:44.390467 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:01:44.391822 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:01:44.391829 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:01:44.392014 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:44.392055 (+ 41us) client_negotiation.cc:769] Sending connection context
0807 23:01:44.392828 (+ 773us) client_negotiation.cc:240] Negotiation successful
0807 23:01:44.392988 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":939,"mutex_wait_us":2,"thread_start_us":755,"threads_started":1}
I20260807 23:01:44.394304 26993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f80ea6e3659494f9a613061a6641890" candidate_uuid: "01e8e20334af414bbab2f6edcd49017a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34cb6cda69a14e70a533dbdb9c9c3542" is_pre_election: true
I20260807 23:01:44.394819 26993 raft_consensus.cc:2379] T 1f80ea6e3659494f9a613061a6641890 P 34cb6cda69a14e70a533dbdb9c9c3542 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 01e8e20334af414bbab2f6edcd49017a in current term 1: Already voted for candidate 34cb6cda69a14e70a533dbdb9c9c3542 in this term.
I20260807 23:01:44.395852 26780 raft_consensus.cc:2454] T 1f80ea6e3659494f9a613061a6641890 P 01e8e20334af414bbab2f6edcd49017a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34cb6cda69a14e70a533dbdb9c9c3542 in term 1.
I20260807 23:01:44.396286 26896 leader_election.cc:302] T 1f80ea6e3659494f9a613061a6641890 P 34cb6cda69a14e70a533dbdb9c9c3542 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01e8e20334af414bbab2f6edcd49017a, 34cb6cda69a14e70a533dbdb9c9c3542; no voters:
I20260807 23:01:44.396412 27308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:44.385408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.3:44197 (local address 127.25.94.1:60951)
0807 23:01:44.389141 (+ 3733us) negotiation.cc:107] Waiting for socket to connect
0807 23:01:44.389161 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 23:01:44.389375 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 23:01:44.389735 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 23:01:44.389740 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 23:01:44.389771 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 23:01:44.390173 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:01:44.390185 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:01:44.391591 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:01:44.391595 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:01:44.393726 (+ 2131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:01:44.393743 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:01:44.395635 (+ 1892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:01:44.395641 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:01:44.395816 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:44.395855 (+ 39us) client_negotiation.cc:769] Sending connection context
0807 23:01:44.395951 (+ 96us) client_negotiation.cc:240] Negotiation successful
0807 23:01:44.396083 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3585,"thread_start_us":3356,"threads_started":1}
I20260807 23:01:44.396442 27283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:44.385770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.94.1:60951 (local address 127.25.94.3:44197)
0807 23:01:44.385935 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0807 23:01:44.385940 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 23:01:44.389265 (+ 3325us) server_negotiation.cc:407] Connection header received
0807 23:01:44.389521 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:01:44.389529 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:01:44.389606 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:01:44.389709 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 23:01:44.390356 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:44.391439 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:44.393970 (+ 2531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:44.395196 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:44.395330 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:44.395456 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0807 23:01:44.396003 (+ 547us) server_negotiation.cc:299] Negotiation successful
0807 23:01:44.396158 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260807 23:01:44.397352 27301 leader_election.cc:288] T 1f80ea6e3659494f9a613061a6641890 P 34cb6cda69a14e70a533dbdb9c9c3542 [CANDIDATE]: Term 1 election: Requested vote from peers 8245571e74574a0f91b392ac6dc72300 (127.25.94.3:44197), 01e8e20334af414bbab2f6edcd49017a (127.25.94.1:36629)
I20260807 23:01:44.397462 27182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f80ea6e3659494f9a613061a6641890" candidate_uuid: "01e8e20334af414bbab2f6edcd49017a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8245571e74574a0f91b392ac6dc72300" is_pre_election: true
I20260807 23:01:44.397724 27301 raft_consensus.cc:2781] T 1f80ea6e3659494f9a613061a6641890 P 34cb6cda69a14e70a533dbdb9c9c3542 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 23:01:44.398106 27301 raft_consensus.cc:686] T 1f80ea6e3659494f9a613061a6641890 P 34cb6cda69a14e70a533dbdb9c9c3542 [term 1 LEADER]: Becoming Leader. State: Replica: 34cb6cda69a14e70a533dbdb9c9c3542, State: Running, Role: LEADER
I20260807 23:01:44.398239 27183 raft_consensus.cc:2454] T 1f80ea6e3659494f9a613061a6641890 P 8245571e74574a0f91b392ac6dc72300 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34cb6cda69a14e70a533dbdb9c9c3542 in term 1.
I20260807 23:01:44.398516 26669 leader_election.cc:302] T 1f80ea6e3659494f9a613061a6641890 P 01e8e20334af414bbab2f6edcd49017a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01e8e20334af414bbab2f6edcd49017a; no voters: 34cb6cda69a14e70a533dbdb9c9c3542, 8245571e74574a0f91b392ac6dc72300
I20260807 23:01:44.398954 27303 raft_consensus.cc:2726] T 1f80ea6e3659494f9a613061a6641890 P 01e8e20334af414bbab2f6edcd49017a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 23:01:44.398957 27301 consensus_queue.cc:237] T 1f80ea6e3659494f9a613061a6641890 P 34cb6cda69a14e70a533dbdb9c9c3542 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34cb6cda69a14e70a533dbdb9c9c3542" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 36589 } } peers { permanent_uuid: "8245571e74574a0f91b392ac6dc72300" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 44197 } } peers { permanent_uuid: "01e8e20334af414bbab2f6edcd49017a" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 36629 } }
I20260807 23:01:44.403695 26547 catalog_manager.cc:5495] T 1f80ea6e3659494f9a613061a6641890 P 34cb6cda69a14e70a533dbdb9c9c3542 reported cstate change: term changed from 0 to 1, leader changed from <none> to 34cb6cda69a14e70a533dbdb9c9c3542 (127.25.94.2). New cstate: current_term: 1 leader_uuid: "34cb6cda69a14e70a533dbdb9c9c3542" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34cb6cda69a14e70a533dbdb9c9c3542" member_type: VOTER last_known_addr { host: "127.25.94.2" port: 36589 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8245571e74574a0f91b392ac6dc72300" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 44197 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "01e8e20334af414bbab2f6edcd49017a" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 36629 } health_report { overall_health: UNKNOWN } } }
I20260807 23:01:44.441470 27281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:44.435952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51868 (local address 127.25.94.2:36589)
0807 23:01:44.436114 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0807 23:01:44.436118 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 23:01:44.436137 (+ 19us) server_negotiation.cc:407] Connection header received
0807 23:01:44.436320 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:01:44.436354 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:01:44.436460 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:01:44.436592 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 23:01:44.436966 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:44.437950 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:44.439340 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:44.439799 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:44.439906 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:44.440360 (+ 454us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 23:01:44.440867 (+ 507us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 23:01:44.440979 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0807 23:01:44.441056 (+ 77us) server_negotiation.cc:299] Negotiation successful
0807 23:01:44.441122 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260807 23:01:44.449904 27183 raft_consensus.cc:1261] T 1f80ea6e3659494f9a613061a6641890 P 8245571e74574a0f91b392ac6dc72300 [term 1 FOLLOWER]: Refusing update from remote peer 34cb6cda69a14e70a533dbdb9c9c3542: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 23:01:44.449960 26780 raft_consensus.cc:1261] T 1f80ea6e3659494f9a613061a6641890 P 01e8e20334af414bbab2f6edcd49017a [term 1 FOLLOWER]: Refusing update from remote peer 34cb6cda69a14e70a533dbdb9c9c3542: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 23:01:44.450809 27309 consensus_queue.cc:1035] T 1f80ea6e3659494f9a613061a6641890 P 34cb6cda69a14e70a533dbdb9c9c3542 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01e8e20334af414bbab2f6edcd49017a" member_type: VOTER last_known_addr { host: "127.25.94.1" port: 36629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 23:01:44.451181 27301 consensus_queue.cc:1035] T 1f80ea6e3659494f9a613061a6641890 P 34cb6cda69a14e70a533dbdb9c9c3542 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8245571e74574a0f91b392ac6dc72300" member_type: VOTER last_known_addr { host: "127.25.94.3" port: 44197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 23:01:44.464586 27314 mvcc.cc:205] Tried to move back new op lower bound from 7316044613416083456 to 7316044613222199296. Current Snapshot: MvccSnapshot[applied={T|T < 7316044613416083456}]
I20260807 23:01:44.478083 26549 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35414:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 23:01:44.478386 26549 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35414:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 23:01:44.482191 26549 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b9a26476b8024b01949f7f1297c67df9: Sending DeleteTablet for 3 replicas of tablet 1f80ea6e3659494f9a613061a6641890
I20260807 23:01:44.483546 26962 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f80ea6e3659494f9a613061a6641890 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 23:01:44 UTC) from {username='slave'} at 127.0.0.1:51866
I20260807 23:01:44.483697 26748 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f80ea6e3659494f9a613061a6641890 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 23:01:44 UTC) from {username='slave'} at 127.0.0.1:55538
I20260807 23:01:44.483758 27160 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f80ea6e3659494f9a613061a6641890 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 23:01:44 UTC) from {username='slave'} at 127.0.0.1:44788
I20260807 23:01:44.485690 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 26645
I20260807 23:01:44.496603 27325 tablet_replica.cc:330] T 1f80ea6e3659494f9a613061a6641890 P 34cb6cda69a14e70a533dbdb9c9c3542: stopping tablet replica
I20260807 23:01:44.497421 27325 raft_consensus.cc:2229] T 1f80ea6e3659494f9a613061a6641890 P 34cb6cda69a14e70a533dbdb9c9c3542 [term 1 LEADER]: Raft consensus shutting down.
I20260807 23:01:44.498469 27325 raft_consensus.cc:2258] T 1f80ea6e3659494f9a613061a6641890 P 34cb6cda69a14e70a533dbdb9c9c3542 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 23:01:44.501012 27325 ts_tablet_manager.cc:1905] T 1f80ea6e3659494f9a613061a6641890 P 34cb6cda69a14e70a533dbdb9c9c3542: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 23:01:44.512018 27325 ts_tablet_manager.cc:1918] T 1f80ea6e3659494f9a613061a6641890 P 34cb6cda69a14e70a533dbdb9c9c3542: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 23:01:44.512269 27327 tablet_replica.cc:330] T 1f80ea6e3659494f9a613061a6641890 P 01e8e20334af414bbab2f6edcd49017a: stopping tablet replica
I20260807 23:01:44.512481 27325 log.cc:1196] T 1f80ea6e3659494f9a613061a6641890 P 34cb6cda69a14e70a533dbdb9c9c3542: Deleting WAL directory at /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786143361779964-25976-0/minicluster-data/ts-1/wal/wals/1f80ea6e3659494f9a613061a6641890
I20260807 23:01:44.512883 27327 raft_consensus.cc:2229] T 1f80ea6e3659494f9a613061a6641890 P 01e8e20334af414bbab2f6edcd49017a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 23:01:44.513391 27325 ts_tablet_manager.cc:1939] T 1f80ea6e3659494f9a613061a6641890 P 34cb6cda69a14e70a533dbdb9c9c3542: Deleting consensus metadata
I20260807 23:01:44.514372 27327 raft_consensus.cc:2258] T 1f80ea6e3659494f9a613061a6641890 P 01e8e20334af414bbab2f6edcd49017a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 23:01:44.516405 26527 catalog_manager.cc:4841] TS 34cb6cda69a14e70a533dbdb9c9c3542 (127.25.94.2:36589): tablet 1f80ea6e3659494f9a613061a6641890 (table test-table [id=a42fac07ca78480ca6a1cb548edf8b65]) successfully deleted
I20260807 23:01:44.518162 27327 ts_tablet_manager.cc:1905] T 1f80ea6e3659494f9a613061a6641890 P 01e8e20334af414bbab2f6edcd49017a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 23:01:44.521700 27328 tablet_replica.cc:330] T 1f80ea6e3659494f9a613061a6641890 P 8245571e74574a0f91b392ac6dc72300: stopping tablet replica
I20260807 23:01:44.522282 27328 raft_consensus.cc:2229] T 1f80ea6e3659494f9a613061a6641890 P 8245571e74574a0f91b392ac6dc72300 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 23:01:44.522882 27328 raft_consensus.cc:2258] T 1f80ea6e3659494f9a613061a6641890 P 8245571e74574a0f91b392ac6dc72300 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 23:01:44.525390 27284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:44.487068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55546 (local address 127.25.94.1:36629)
0807 23:01:44.487256 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0807 23:01:44.487263 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 23:01:44.513815 (+ 26552us) server_negotiation.cc:407] Connection header received
0807 23:01:44.514036 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:01:44.514042 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:01:44.514140 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:01:44.514243 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:01:44.515016 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:44.516169 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:44.519812 (+ 3643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:44.520835 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:44.520967 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:44.522645 (+ 1678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:01:44.522657 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:01:44.522674 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 23:01:44.523478 (+ 804us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:01:44.523547 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 23:01:44.523572 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:01:44.523668 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0807 23:01:44.524823 (+ 1155us) server_negotiation.cc:299] Negotiation successful
0807 23:01:44.524958 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260807 23:01:44.525738 27328 ts_tablet_manager.cc:1905] T 1f80ea6e3659494f9a613061a6641890 P 8245571e74574a0f91b392ac6dc72300: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 23:01:44.526768 26852 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 23:01:44.535406 27328 ts_tablet_manager.cc:1918] T 1f80ea6e3659494f9a613061a6641890 P 8245571e74574a0f91b392ac6dc72300: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 23:01:44.535698 27328 log.cc:1196] T 1f80ea6e3659494f9a613061a6641890 P 8245571e74574a0f91b392ac6dc72300: Deleting WAL directory at /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786143361779964-25976-0/minicluster-data/ts-2/wal/wals/1f80ea6e3659494f9a613061a6641890
I20260807 23:01:44.536295 27328 ts_tablet_manager.cc:1939] T 1f80ea6e3659494f9a613061a6641890 P 8245571e74574a0f91b392ac6dc72300: Deleting consensus metadata
I20260807 23:01:44.538439 26527 catalog_manager.cc:4841] TS 8245571e74574a0f91b392ac6dc72300 (127.25.94.3:44197): tablet 1f80ea6e3659494f9a613061a6641890 (table test-table [id=a42fac07ca78480ca6a1cb548edf8b65]) successfully deleted
I20260807 23:01:44.646347 27327 ts_tablet_manager.cc:1918] T 1f80ea6e3659494f9a613061a6641890 P 01e8e20334af414bbab2f6edcd49017a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 23:01:44.646644 27327 log.cc:1196] T 1f80ea6e3659494f9a613061a6641890 P 01e8e20334af414bbab2f6edcd49017a: Deleting WAL directory at /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786143361779964-25976-0/minicluster-data/ts-0/wal/wals/1f80ea6e3659494f9a613061a6641890
I20260807 23:01:44.647217 27327 ts_tablet_manager.cc:1939] T 1f80ea6e3659494f9a613061a6641890 P 01e8e20334af414bbab2f6edcd49017a: Deleting consensus metadata
I20260807 23:01:44.648909 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 26645
I20260807 23:01:44.648963 26527 catalog_manager.cc:4841] TS 01e8e20334af414bbab2f6edcd49017a (127.25.94.1:36629): tablet 1f80ea6e3659494f9a613061a6641890 (table test-table [id=a42fac07ca78480ca6a1cb548edf8b65]) successfully deleted
I20260807 23:01:44.678155 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 26865
I20260807 23:01:44.685731 27281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:44.679127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51884 (local address 127.25.94.2:36589)
0807 23:01:44.679308 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0807 23:01:44.679316 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 23:01:44.679337 (+ 21us) server_negotiation.cc:407] Connection header received
0807 23:01:44.679585 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:01:44.679591 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:01:44.679696 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:01:44.679840 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:01:44.680234 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:44.681006 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:44.682366 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:44.682826 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:44.682976 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:44.684130 (+ 1154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:01:44.684140 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:01:44.684155 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 23:01:44.684806 (+ 651us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:01:44.684862 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 23:01:44.684883 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:01:44.685005 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0807 23:01:44.685149 (+ 144us) server_negotiation.cc:299] Negotiation successful
0807 23:01:44.685392 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260807 23:01:44.686429 27060 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 23:01:44.800580 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 26865
I20260807 23:01:44.829068 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 27072
I20260807 23:01:44.836783 27283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:44.830010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44790 (local address 127.25.94.3:44197)
0807 23:01:44.830200 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0807 23:01:44.830208 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 23:01:44.830233 (+ 25us) server_negotiation.cc:407] Connection header received
0807 23:01:44.830482 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:01:44.830488 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:01:44.830587 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:01:44.830701 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:01:44.831170 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:44.831966 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:44.833647 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:44.834074 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:44.834239 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:44.835386 (+ 1147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:01:44.835395 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:01:44.835411 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 23:01:44.836013 (+ 602us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:01:44.836063 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 23:01:44.836082 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:01:44.836215 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0807 23:01:44.836357 (+ 142us) server_negotiation.cc:299] Negotiation successful
0807 23:01:44.836460 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260807 23:01:44.837590 27253 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 23:01:44.927799 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 27072
I20260807 23:01:44.952775 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 26502
I20260807 23:01:44.953991 26597 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 23:01:45.064384 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 26502
2026-08-07T23:01:45Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (3215 ms)
[----------] 6 tests from EncryptionPolicyTest (24470 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786143361779964-25976-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 23:01:45 dist-test-slave-z8x9 krb5kdc[27340](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 23:01:45 dist-test-slave-z8x9 krb5kdc[27340](info): set up 2 sockets
Aug 07 23:01:45 dist-test-slave-z8x9 krb5kdc[27340](info): commencing operation
krb5kdc: starting...
W20260807 23:01:46.462003 25976 mini_kdc.cc:121] Time spent starting KDC: real 1.340s user 0.009s sys 0.038s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786143361779964-25976-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786143361779964-25976-0/krb5kdc/test-user.keytab.
Aug 07 23:01:46 dist-test-slave-z8x9 krb5kdc[27340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143706, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T23:01:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T23:01:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786143361779964-25976-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786143361779964-25976-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 23:01:46.785364 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786143361779964-25976-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786143361779964-25976-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786143361779964-25976-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41219
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786143361779964-25976-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41219
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786143361779964-25976-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786143361779964-25976-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786143361779964-25976-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 23:01:47.138306 27363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 23:01:47.138614 27363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 23:01:47.138712 27363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 23:01:47.146224 27363 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 23:01:47.146359 27363 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 23:01:47.146409 27363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 23:01:47.146473 27363 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 23:01:47.146519 27363 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 23:01:47.156895 27363 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786143361779964-25976-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786143361779964-25976-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41219
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786143361779964-25976-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41219
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786143361779964-25976-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 23:01:47.158363 27363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 23:01:47.159700 27363 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 23:01:47.172278 27373 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 23:01:47.172278 27372 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 23:01:47.172748 27375 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 23:01:47.173107 27363 server_base.cc:1034] running on GCE node
I20260807 23:01:47.174075 27363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 23:01:47.175676 27363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 23:01:47.176949 27363 hybrid_clock.cc:648] HybridClock initialized: now 1786143707176914 us; error 59 us; skew 500 ppm
I20260807 23:01:47.177429 27363 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 23:01:47 dist-test-slave-z8x9 krb5kdc[27340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143707, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 23:01:47.181016 27363 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 23:01:47.183063 27363 webserver.cc:460] Webserver started at http://127.0.0.1:33747/ using document root <none> and password file <none>
I20260807 23:01:47.184342 27363 fs_manager.cc:359] Metadata directory not provided
I20260807 23:01:47.184748 27363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 23:01:47.185395 27363 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 23:01:47.189369 27363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786143361779964-25976-0/minicluster-data/master-0/data/instance:
uuid: "f958ce65707749dbb90d70fb3dc4ca99"
format_stamp: "Formatted at 2026-08-07 23:01:47 on dist-test-slave-z8x9"
server_key: "3162b68d957a528725aaa5168db7bd93"
server_key_iv: "aa3ca18a090a309ffce472acc65f79c2"
server_key_version: "encryptionkey@0"
I20260807 23:01:47.191038 27363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786143361779964-25976-0/minicluster-data/master-0/wal/instance:
uuid: "f958ce65707749dbb90d70fb3dc4ca99"
format_stamp: "Formatted at 2026-08-07 23:01:47 on dist-test-slave-z8x9"
server_key: "3162b68d957a528725aaa5168db7bd93"
server_key_iv: "aa3ca18a090a309ffce472acc65f79c2"
server_key_version: "encryptionkey@0"
I20260807 23:01:47.197710 27363 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260807 23:01:47.202020 27383 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 23:01:47.203933 27363 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260807 23:01:47.204188 27363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786143361779964-25976-0/minicluster-data/master-0/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786143361779964-25976-0/minicluster-data/master-0/wal
uuid: "f958ce65707749dbb90d70fb3dc4ca99"
format_stamp: "Formatted at 2026-08-07 23:01:47 on dist-test-slave-z8x9"
server_key: "3162b68d957a528725aaa5168db7bd93"
server_key_iv: "aa3ca18a090a309ffce472acc65f79c2"
server_key_version: "encryptionkey@0"
I20260807 23:01:47.204453 27363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786143361779964-25976-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786143361779964-25976-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786143361779964-25976-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 23:01:47.234943 27363 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 23:01:47.239892 27363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 23:01:47.240211 27363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 23:01:47.261281 27363 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41219
I20260807 23:01:47.261287 27461 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41219 every 8 connection(s)
I20260807 23:01:47.262880 27363 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
I20260807 23:01:47.266743 27462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 23:01:47.268811 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 27363
I20260807 23:01:47.269068 25976 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786143361779964-25976-0/minicluster-data/master-0/wal/instance
I20260807 23:01:47.269917 25976 external_mini_cluster.cc:1428] Setting key 1b489ca7bf5078ad0f808f3ca79d97b9
I20260807 23:01:47.280125 27462 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f958ce65707749dbb90d70fb3dc4ca99: Bootstrap starting.
I20260807 23:01:47.284312 27462 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f958ce65707749dbb90d70fb3dc4ca99: Neither blocks nor log segments found. Creating new log.
I20260807 23:01:47.285642 27462 log.cc:824] T 00000000000000000000000000000000 P f958ce65707749dbb90d70fb3dc4ca99: Log is configured to *not* fsync() on all Append() calls
Aug 07 23:01:47 dist-test-slave-z8x9 krb5kdc[27340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143706, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260807 23:01:47.288753 27462 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f958ce65707749dbb90d70fb3dc4ca99: No bootstrap required, opened a new log
I20260807 23:01:47.295002 27467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:47.272124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56823 (local address 127.0.0.1:41219)
0807 23:01:47.273010 (+ 886us) server_negotiation.cc:206] Beginning negotiation
0807 23:01:47.273022 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 23:01:47.273070 (+ 48us) server_negotiation.cc:407] Connection header received
0807 23:01:47.274836 (+ 1766us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:01:47.274857 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:01:47.275239 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:01:47.275562 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:01:47.277448 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:47.278808 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:47.280668 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:47.281493 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:47.281646 (+ 153us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:47.287074 (+ 5428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:01:47.287083 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:01:47.287099 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 23:01:47.287143 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:01:47.290441 (+ 3298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:01:47.291331 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:01:47.291337 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:01:47.291347 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:01:47.291513 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:01:47.292060 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:01:47.292072 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:01:47.292075 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:01:47.292619 (+ 544us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:01:47.292977 (+ 358us) server_negotiation.cc:1035] Waiting for connection context
0807 23:01:47.293556 (+ 579us) server_negotiation.cc:299] Negotiation successful
0807 23:01:47.293919 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":669,"thread_start_us":384,"threads_started":1}
I20260807 23:01:47.297102 27462 raft_consensus.cc:348] T 00000000000000000000000000000000 P f958ce65707749dbb90d70fb3dc4ca99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f958ce65707749dbb90d70fb3dc4ca99" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41219 } }
I20260807 23:01:47.297688 27462 raft_consensus.cc:374] T 00000000000000000000000000000000 P f958ce65707749dbb90d70fb3dc4ca99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 23:01:47.297797 27462 raft_consensus.cc:729] T 00000000000000000000000000000000 P f958ce65707749dbb90d70fb3dc4ca99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f958ce65707749dbb90d70fb3dc4ca99, State: Initialized, Role: FOLLOWER
I20260807 23:01:47.298491 27462 consensus_queue.cc:260] T 00000000000000000000000000000000 P f958ce65707749dbb90d70fb3dc4ca99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f958ce65707749dbb90d70fb3dc4ca99" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41219 } }
I20260807 23:01:47.298771 27462 raft_consensus.cc:388] T 00000000000000000000000000000000 P f958ce65707749dbb90d70fb3dc4ca99 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 23:01:47.298947 27462 raft_consensus.cc:482] T 00000000000000000000000000000000 P f958ce65707749dbb90d70fb3dc4ca99 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 23:01:47.299132 27462 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f958ce65707749dbb90d70fb3dc4ca99 [term 0 FOLLOWER]: Advancing to term 1
I20260807 23:01:47.299324 25976 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 23:01:47.301982 27462 raft_consensus.cc:504] T 00000000000000000000000000000000 P f958ce65707749dbb90d70fb3dc4ca99 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f958ce65707749dbb90d70fb3dc4ca99" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41219 } }
I20260807 23:01:47.302839 27462 leader_election.cc:302] T 00000000000000000000000000000000 P f958ce65707749dbb90d70fb3dc4ca99 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f958ce65707749dbb90d70fb3dc4ca99; no voters:
I20260807 23:01:47.303570 27462 leader_election.cc:288] T 00000000000000000000000000000000 P f958ce65707749dbb90d70fb3dc4ca99 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 23:01:47.303684 27478 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f958ce65707749dbb90d70fb3dc4ca99 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 23:01:47.306460 27462 sys_catalog.cc:564] T 00000000000000000000000000000000 P f958ce65707749dbb90d70fb3dc4ca99 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 23:01:47.310735 27478 raft_consensus.cc:686] T 00000000000000000000000000000000 P f958ce65707749dbb90d70fb3dc4ca99 [term 1 LEADER]: Becoming Leader. State: Replica: f958ce65707749dbb90d70fb3dc4ca99, State: Running, Role: LEADER
I20260807 23:01:47.311669 27478 consensus_queue.cc:237] T 00000000000000000000000000000000 P f958ce65707749dbb90d70fb3dc4ca99 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f958ce65707749dbb90d70fb3dc4ca99" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41219 } }
I20260807 23:01:47.318847 27480 sys_catalog.cc:455] T 00000000000000000000000000000000 P f958ce65707749dbb90d70fb3dc4ca99 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f958ce65707749dbb90d70fb3dc4ca99. Latest consensus state: current_term: 1 leader_uuid: "f958ce65707749dbb90d70fb3dc4ca99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f958ce65707749dbb90d70fb3dc4ca99" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41219 } } }
I20260807 23:01:47.318922 27479 sys_catalog.cc:455] T 00000000000000000000000000000000 P f958ce65707749dbb90d70fb3dc4ca99 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f958ce65707749dbb90d70fb3dc4ca99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f958ce65707749dbb90d70fb3dc4ca99" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41219 } } }
I20260807 23:01:47.319494 27480 sys_catalog.cc:458] T 00000000000000000000000000000000 P f958ce65707749dbb90d70fb3dc4ca99 [sys.catalog]: This master's current role is: LEADER
I20260807 23:01:47.319522 27479 sys_catalog.cc:458] T 00000000000000000000000000000000 P f958ce65707749dbb90d70fb3dc4ca99 [sys.catalog]: This master's current role is: LEADER
I20260807 23:01:47.320570 27487 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 23:01:47.322952 27467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:47.305245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46301 (local address 127.0.0.1:41219)
0807 23:01:47.305412 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0807 23:01:47.305418 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 23:01:47.305677 (+ 259us) server_negotiation.cc:407] Connection header received
0807 23:01:47.305990 (+ 313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:01:47.305996 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:01:47.306090 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:01:47.306208 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:01:47.307869 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:47.308954 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:47.310636 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:47.311265 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:47.311412 (+ 147us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:47.314614 (+ 3202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:01:47.314620 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:01:47.314625 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 23:01:47.314677 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:01:47.318164 (+ 3487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:01:47.319371 (+ 1207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:01:47.319377 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:01:47.319380 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:01:47.319547 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:01:47.320255 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:01:47.320260 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:01:47.320263 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:01:47.320740 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:01:47.321026 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0807 23:01:47.322368 (+ 1342us) server_negotiation.cc:299] Negotiation successful
0807 23:01:47.322566 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260807 23:01:47.327672 27487 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 23:01:47.338028 27487 catalog_manager.cc:1347] Generated new cluster ID: 0e4e6b7be6994a89bc47967a49ff47e9
I20260807 23:01:47.338176 27487 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 23:01:47.359145 27487 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 23:01:47.361106 27487 catalog_manager.cc:1504] Loading token signing keys...
I20260807 23:01:47.371507 27487 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f958ce65707749dbb90d70fb3dc4ca99: Generated new TSK 0
I20260807 23:01:47.372524 27487 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 23:01:47.388754 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 27363
I20260807 23:01:47.390969 27449 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 23:01:47.484565 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 27363
W20260807 23:01:47.509032 27336 connection.cc:535] client connection to 127.0.0.1:41219 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41219): Connection reset by peer (error 104)
2026-08-07T23:01:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2407 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786143361779964-25976-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 23:01:47 dist-test-slave-z8x9 krb5kdc[27506](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 23:01:47 dist-test-slave-z8x9 krb5kdc[27506](info): set up 2 sockets
Aug 07 23:01:47 dist-test-slave-z8x9 krb5kdc[27506](info): commencing operation
krb5kdc: starting...
W20260807 23:01:48.876751 25976 mini_kdc.cc:121] Time spent starting KDC: real 1.348s user 0.000s sys 0.044s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786143361779964-25976-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786143361779964-25976-0/krb5kdc/test-user.keytab.
Aug 07 23:01:49 dist-test-slave-z8x9 krb5kdc[27506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143709, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T23:01:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T23:01:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786143361779964-25976-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786143361779964-25976-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 23:01:49.215011 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786143361779964-25976-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786143361779964-25976-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786143361779964-25976-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39145
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786143361779964-25976-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39145
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786143361779964-25976-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786143361779964-25976-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786143361779964-25976-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 23:01:49.563596 27522 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 23:01:49.563982 27522 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 23:01:49.564157 27522 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 23:01:49.571539 27522 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 23:01:49.571717 27522 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 23:01:49.571828 27522 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 23:01:49.571938 27522 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 23:01:49.572022 27522 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 23:01:49.582271 27522 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786143361779964-25976-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786143361779964-25976-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39145
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786143361779964-25976-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39145
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786143361779964-25976-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 23:01:49.583674 27522 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 23:01:49.585029 27522 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 23:01:49.597239 27527 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 23:01:49.597200 27529 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 23:01:49.597241 27531 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 23:01:49.598304 27522 server_base.cc:1034] running on GCE node
I20260807 23:01:49.599124 27522 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 23:01:49.600539 27522 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 23:01:49.601747 27522 hybrid_clock.cc:648] HybridClock initialized: now 1786143709601703 us; error 38 us; skew 500 ppm
I20260807 23:01:49.602056 27522 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 23:01:49 dist-test-slave-z8x9 krb5kdc[27506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143709, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 23:01:49.609929 27522 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 23:01:49.611690 27522 webserver.cc:460] Webserver started at http://127.0.0.1:38357/ using document root <none> and password file <none>
I20260807 23:01:49.612458 27522 fs_manager.cc:359] Metadata directory not provided
I20260807 23:01:49.612613 27522 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 23:01:49.612948 27522 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 23:01:49.616122 27522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786143361779964-25976-0/minicluster-data/master-0/data/instance:
uuid: "dd6ae4929b5e460ca5c5475d5d160e42"
format_stamp: "Formatted at 2026-08-07 23:01:49 on dist-test-slave-z8x9"
server_key: "91d7c82cc5b94a930441463e3f6bfaa7"
server_key_iv: "aac24393b2acb156d68de23337f68c44"
server_key_version: "encryptionkey@0"
I20260807 23:01:49.616957 27522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786143361779964-25976-0/minicluster-data/master-0/wal/instance:
uuid: "dd6ae4929b5e460ca5c5475d5d160e42"
format_stamp: "Formatted at 2026-08-07 23:01:49 on dist-test-slave-z8x9"
server_key: "91d7c82cc5b94a930441463e3f6bfaa7"
server_key_iv: "aac24393b2acb156d68de23337f68c44"
server_key_version: "encryptionkey@0"
I20260807 23:01:49.622539 27522 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260807 23:01:49.626292 27542 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 23:01:49.627926 27522 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260807 23:01:49.628113 27522 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786143361779964-25976-0/minicluster-data/master-0/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786143361779964-25976-0/minicluster-data/master-0/wal
uuid: "dd6ae4929b5e460ca5c5475d5d160e42"
format_stamp: "Formatted at 2026-08-07 23:01:49 on dist-test-slave-z8x9"
server_key: "91d7c82cc5b94a930441463e3f6bfaa7"
server_key_iv: "aac24393b2acb156d68de23337f68c44"
server_key_version: "encryptionkey@0"
I20260807 23:01:49.628324 27522 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786143361779964-25976-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786143361779964-25976-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786143361779964-25976-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 23:01:49.688185 27522 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 23:01:49.693228 27522 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 23:01:49.693564 27522 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 23:01:49.712971 27522 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39145
I20260807 23:01:49.713013 27613 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39145 every 8 connection(s)
I20260807 23:01:49.714398 27522 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
I20260807 23:01:49.718101 27614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 23:01:49.719302 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 27522
I20260807 23:01:49.719556 25976 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786143361779964-25976-0/minicluster-data/master-0/wal/instance
I20260807 23:01:49.720171 25976 external_mini_cluster.cc:1428] Setting key bbfde206ef9360b92e6b6c141541d08d
I20260807 23:01:49.732160 27614 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd6ae4929b5e460ca5c5475d5d160e42: Bootstrap starting.
I20260807 23:01:49.733750 27618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:49.722009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60191 (local address 127.0.0.1:39145)
0807 23:01:49.722857 (+ 848us) server_negotiation.cc:206] Beginning negotiation
0807 23:01:49.722868 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 23:01:49.722907 (+ 39us) server_negotiation.cc:407] Connection header received
0807 23:01:49.724493 (+ 1586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:01:49.724516 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:01:49.724890 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:01:49.725273 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:01:49.725852 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:49.727110 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:49.729062 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:49.729859 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:49.729989 (+ 130us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:49.731313 (+ 1324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:01:49.731321 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:01:49.731333 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 23:01:49.731900 (+ 567us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:01:49.731980 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 23:01:49.732004 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:01:49.732120 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0807 23:01:49.732478 (+ 358us) server_negotiation.cc:299] Negotiation successful
0807 23:01:49.732653 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":620,"thread_start_us":395,"threads_started":1}
I20260807 23:01:49.735755 27614 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dd6ae4929b5e460ca5c5475d5d160e42: Neither blocks nor log segments found. Creating new log.
I20260807 23:01:49.737089 27614 log.cc:824] T 00000000000000000000000000000000 P dd6ae4929b5e460ca5c5475d5d160e42: Log is configured to *not* fsync() on all Append() calls
I20260807 23:01:49.737773 25976 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 23:01:49.739930 27614 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd6ae4929b5e460ca5c5475d5d160e42: No bootstrap required, opened a new log
I20260807 23:01:49.749341 27618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:49.742371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51667 (local address 127.0.0.1:39145)
0807 23:01:49.742592 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0807 23:01:49.742601 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 23:01:49.743031 (+ 430us) server_negotiation.cc:407] Connection header received
0807 23:01:49.743186 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:01:49.743190 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:01:49.743248 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:01:49.743343 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:01:49.743954 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:49.744700 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:49.746673 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:49.747130 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:49.747229 (+ 99us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:49.748504 (+ 1275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:01:49.748509 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:01:49.748512 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 23:01:49.748529 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:01:49.748581 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 23:01:49.748597 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:01:49.748683 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0807 23:01:49.748921 (+ 238us) server_negotiation.cc:299] Negotiation successful
0807 23:01:49.748989 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260807 23:01:49.749891 27614 raft_consensus.cc:348] T 00000000000000000000000000000000 P dd6ae4929b5e460ca5c5475d5d160e42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd6ae4929b5e460ca5c5475d5d160e42" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39145 } }
I20260807 23:01:49.750301 27614 raft_consensus.cc:374] T 00000000000000000000000000000000 P dd6ae4929b5e460ca5c5475d5d160e42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 23:01:49.750432 27614 raft_consensus.cc:729] T 00000000000000000000000000000000 P dd6ae4929b5e460ca5c5475d5d160e42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd6ae4929b5e460ca5c5475d5d160e42, State: Initialized, Role: FOLLOWER
I20260807 23:01:49.751044 27614 consensus_queue.cc:260] T 00000000000000000000000000000000 P dd6ae4929b5e460ca5c5475d5d160e42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd6ae4929b5e460ca5c5475d5d160e42" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39145 } }
I20260807 23:01:49.751299 27614 raft_consensus.cc:388] T 00000000000000000000000000000000 P dd6ae4929b5e460ca5c5475d5d160e42 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 23:01:49.751447 27614 raft_consensus.cc:482] T 00000000000000000000000000000000 P dd6ae4929b5e460ca5c5475d5d160e42 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 23:01:49.751636 27614 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dd6ae4929b5e460ca5c5475d5d160e42 [term 0 FOLLOWER]: Advancing to term 1
I20260807 23:01:49.753925 27614 raft_consensus.cc:504] T 00000000000000000000000000000000 P dd6ae4929b5e460ca5c5475d5d160e42 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd6ae4929b5e460ca5c5475d5d160e42" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39145 } }
I20260807 23:01:49.754441 27614 leader_election.cc:302] T 00000000000000000000000000000000 P dd6ae4929b5e460ca5c5475d5d160e42 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dd6ae4929b5e460ca5c5475d5d160e42; no voters:
I20260807 23:01:49.755081 27614 leader_election.cc:288] T 00000000000000000000000000000000 P dd6ae4929b5e460ca5c5475d5d160e42 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 23:01:49.755271 27630 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dd6ae4929b5e460ca5c5475d5d160e42 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 23:01:49.757082 27614 sys_catalog.cc:564] T 00000000000000000000000000000000 P dd6ae4929b5e460ca5c5475d5d160e42 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 23:01:49.757472 27630 raft_consensus.cc:686] T 00000000000000000000000000000000 P dd6ae4929b5e460ca5c5475d5d160e42 [term 1 LEADER]: Becoming Leader. State: Replica: dd6ae4929b5e460ca5c5475d5d160e42, State: Running, Role: LEADER
I20260807 23:01:49.757997 27630 consensus_queue.cc:237] T 00000000000000000000000000000000 P dd6ae4929b5e460ca5c5475d5d160e42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd6ae4929b5e460ca5c5475d5d160e42" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39145 } }
I20260807 23:01:49.763710 27632 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd6ae4929b5e460ca5c5475d5d160e42 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dd6ae4929b5e460ca5c5475d5d160e42" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd6ae4929b5e460ca5c5475d5d160e42" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39145 } } }
I20260807 23:01:49.763607 27635 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd6ae4929b5e460ca5c5475d5d160e42 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dd6ae4929b5e460ca5c5475d5d160e42. Latest consensus state: current_term: 1 leader_uuid: "dd6ae4929b5e460ca5c5475d5d160e42" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd6ae4929b5e460ca5c5475d5d160e42" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39145 } } }
I20260807 23:01:49.764276 27632 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd6ae4929b5e460ca5c5475d5d160e42 [sys.catalog]: This master's current role is: LEADER
I20260807 23:01:49.764292 27635 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd6ae4929b5e460ca5c5475d5d160e42 [sys.catalog]: This master's current role is: LEADER
I20260807 23:01:49.766646 27655 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 23:01:49.772702 27655 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 23:01:49.782835 27655 catalog_manager.cc:1347] Generated new cluster ID: db905829347f4705af2958026b91f9b0
I20260807 23:01:49.783035 27655 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 23:01:49.805006 27655 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 23:01:49.806421 27655 catalog_manager.cc:1504] Loading token signing keys...
I20260807 23:01:49.818699 27655 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dd6ae4929b5e460ca5c5475d5d160e42: Generated new TSK 0
I20260807 23:01:49.819828 27655 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 23:01:49.828994 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 27522
W20260807 23:01:49.830366 27606 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:60191
W20260807 23:01:49.831168 25976 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 27522: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 23:01:49.831360 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 27522
W20260807 23:01:49.855989 27503 connection.cc:535] client connection to 127.0.0.1:39145 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39145): Connection reset by peer (error 104)
2026-08-07T23:01:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (2346 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786143361779964-25976-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 23:01:49 dist-test-slave-z8x9 krb5kdc[27664](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 23:01:49 dist-test-slave-z8x9 krb5kdc[27664](info): set up 2 sockets
Aug 07 23:01:49 dist-test-slave-z8x9 krb5kdc[27664](info): commencing operation
krb5kdc: starting...
W20260807 23:01:51.223246 25976 mini_kdc.cc:121] Time spent starting KDC: real 1.348s user 0.000s sys 0.045s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786143361779964-25976-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786143361779964-25976-0/krb5kdc/test-user.keytab.
Aug 07 23:01:51 dist-test-slave-z8x9 krb5kdc[27664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143711, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T23:01:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T23:01:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786143361779964-25976-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786143361779964-25976-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 23:01:51.553261 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786143361779964-25976-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786143361779964-25976-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786143361779964-25976-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33721
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786143361779964-25976-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33721
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786143361779964-25976-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786143361779964-25976-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786143361779964-25976-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 23:01:51.905925 27684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 23:01:51.906234 27684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 23:01:51.906430 27684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 23:01:51.913973 27684 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 23:01:51.914129 27684 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 23:01:51.914229 27684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 23:01:51.914330 27684 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 23:01:51.914407 27684 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 23:01:51.925002 27684 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786143361779964-25976-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786143361779964-25976-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33721
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786143361779964-25976-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33721
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786143361779964-25976-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 23:01:51.926457 27684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 23:01:51.927814 27684 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 23:01:51.940248 27689 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 23:01:51.940296 27694 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 23:01:51.940296 27691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 23:01:51.941028 27684 server_base.cc:1034] running on GCE node
I20260807 23:01:51.941924 27684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 23:01:51.943424 27684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 23:01:51.944701 27684 hybrid_clock.cc:648] HybridClock initialized: now 1786143711944656 us; error 28 us; skew 500 ppm
I20260807 23:01:51.945099 27684 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 23:01:51 dist-test-slave-z8x9 krb5kdc[27664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143711, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 23:01:51.948413 27684 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 23:01:51.950043 27684 webserver.cc:460] Webserver started at http://127.0.0.1:32833/ using document root <none> and password file <none>
I20260807 23:01:51.950762 27684 fs_manager.cc:359] Metadata directory not provided
I20260807 23:01:51.950917 27684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 23:01:51.951249 27684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 23:01:51.954483 27684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786143361779964-25976-0/minicluster-data/master-0/data/instance:
uuid: "677b2c4e232e4423bdf25f46ac843637"
format_stamp: "Formatted at 2026-08-07 23:01:51 on dist-test-slave-z8x9"
server_key: "9c70b9348502317eecddb153c3b9f4b5"
server_key_iv: "27c66fbb3d602826d40d8da4515b5abc"
server_key_version: "encryptionkey@0"
I20260807 23:01:51.955308 27684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786143361779964-25976-0/minicluster-data/master-0/wal/instance:
uuid: "677b2c4e232e4423bdf25f46ac843637"
format_stamp: "Formatted at 2026-08-07 23:01:51 on dist-test-slave-z8x9"
server_key: "9c70b9348502317eecddb153c3b9f4b5"
server_key_iv: "27c66fbb3d602826d40d8da4515b5abc"
server_key_version: "encryptionkey@0"
I20260807 23:01:51.960681 27684 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260807 23:01:51.965269 27705 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 23:01:51.966917 27684 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260807 23:01:51.967101 27684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786143361779964-25976-0/minicluster-data/master-0/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786143361779964-25976-0/minicluster-data/master-0/wal
uuid: "677b2c4e232e4423bdf25f46ac843637"
format_stamp: "Formatted at 2026-08-07 23:01:51 on dist-test-slave-z8x9"
server_key: "9c70b9348502317eecddb153c3b9f4b5"
server_key_iv: "27c66fbb3d602826d40d8da4515b5abc"
server_key_version: "encryptionkey@0"
I20260807 23:01:51.967324 27684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786143361779964-25976-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786143361779964-25976-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786143361779964-25976-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 23:01:51.973558 27684 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 23:01:51.978343 27684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 23:01:51.978617 27684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 23:01:51.998377 27684 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33721
I20260807 23:01:51.998432 27779 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33721 every 8 connection(s)
I20260807 23:01:51.999867 27684 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
I20260807 23:01:52.003839 27783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 23:01:52.005016 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 27684
I20260807 23:01:52.005239 25976 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786143361779964-25976-0/minicluster-data/master-0/wal/instance
I20260807 23:01:52.005764 25976 external_mini_cluster.cc:1428] Setting key b65a931eaf281b54c6f79b79e993de9f
I20260807 23:01:52.014672 27788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:52.007919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54079 (local address 127.0.0.1:33721)
0807 23:01:52.008821 (+ 902us) server_negotiation.cc:206] Beginning negotiation
0807 23:01:52.008833 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 23:01:52.008883 (+ 50us) server_negotiation.cc:407] Connection header received
0807 23:01:52.010633 (+ 1750us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:01:52.010662 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:01:52.011064 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:01:52.011434 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:01:52.012698 (+ 1264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:01:52.012705 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:01:52.012720 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 23:01:52.012738 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:01:52.012803 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 23:01:52.012826 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:01:52.013007 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0807 23:01:52.013121 (+ 114us) server_negotiation.cc:299] Negotiation successful
0807 23:01:52.013529 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":654,"thread_start_us":411,"threads_started":1}
I20260807 23:01:52.018772 27783 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 677b2c4e232e4423bdf25f46ac843637: Bootstrap starting.
I20260807 23:01:52.018795 25976 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 23:01:52.021924 27783 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 677b2c4e232e4423bdf25f46ac843637: Neither blocks nor log segments found. Creating new log.
I20260807 23:01:52.023135 27783 log.cc:824] T 00000000000000000000000000000000 P 677b2c4e232e4423bdf25f46ac843637: Log is configured to *not* fsync() on all Append() calls
I20260807 23:01:52.026373 27788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:52.023348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51179 (local address 127.0.0.1:33721)
0807 23:01:52.023553 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0807 23:01:52.023561 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 23:01:52.024038 (+ 477us) server_negotiation.cc:407] Connection header received
0807 23:01:52.024224 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:01:52.024227 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:01:52.024278 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:01:52.024377 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:01:52.025605 (+ 1228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:01:52.025610 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:01:52.025614 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 23:01:52.025636 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:01:52.025694 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 23:01:52.025716 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:01:52.025860 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0807 23:01:52.026016 (+ 156us) server_negotiation.cc:299] Negotiation successful
0807 23:01:52.026080 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260807 23:01:52.026950 27783 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 677b2c4e232e4423bdf25f46ac843637: No bootstrap required, opened a new log
I20260807 23:01:52.034766 27783 raft_consensus.cc:348] T 00000000000000000000000000000000 P 677b2c4e232e4423bdf25f46ac843637 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "677b2c4e232e4423bdf25f46ac843637" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33721 } }
I20260807 23:01:52.035169 27783 raft_consensus.cc:374] T 00000000000000000000000000000000 P 677b2c4e232e4423bdf25f46ac843637 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 23:01:52.035244 27783 raft_consensus.cc:729] T 00000000000000000000000000000000 P 677b2c4e232e4423bdf25f46ac843637 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 677b2c4e232e4423bdf25f46ac843637, State: Initialized, Role: FOLLOWER
I20260807 23:01:52.035857 27783 consensus_queue.cc:260] T 00000000000000000000000000000000 P 677b2c4e232e4423bdf25f46ac843637 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "677b2c4e232e4423bdf25f46ac843637" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33721 } }
I20260807 23:01:52.036077 27783 raft_consensus.cc:388] T 00000000000000000000000000000000 P 677b2c4e232e4423bdf25f46ac843637 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 23:01:52.036176 27783 raft_consensus.cc:482] T 00000000000000000000000000000000 P 677b2c4e232e4423bdf25f46ac843637 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 23:01:52.036305 27783 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 677b2c4e232e4423bdf25f46ac843637 [term 0 FOLLOWER]: Advancing to term 1
I20260807 23:01:52.038650 27783 raft_consensus.cc:504] T 00000000000000000000000000000000 P 677b2c4e232e4423bdf25f46ac843637 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "677b2c4e232e4423bdf25f46ac843637" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33721 } }
I20260807 23:01:52.039143 27783 leader_election.cc:302] T 00000000000000000000000000000000 P 677b2c4e232e4423bdf25f46ac843637 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 677b2c4e232e4423bdf25f46ac843637; no voters:
I20260807 23:01:52.039867 27783 leader_election.cc:288] T 00000000000000000000000000000000 P 677b2c4e232e4423bdf25f46ac843637 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 23:01:52.040004 27796 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 677b2c4e232e4423bdf25f46ac843637 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 23:01:52.041329 27796 raft_consensus.cc:686] T 00000000000000000000000000000000 P 677b2c4e232e4423bdf25f46ac843637 [term 1 LEADER]: Becoming Leader. State: Replica: 677b2c4e232e4423bdf25f46ac843637, State: Running, Role: LEADER
I20260807 23:01:52.041895 27796 consensus_queue.cc:237] T 00000000000000000000000000000000 P 677b2c4e232e4423bdf25f46ac843637 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "677b2c4e232e4423bdf25f46ac843637" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33721 } }
I20260807 23:01:52.042366 27783 sys_catalog.cc:564] T 00000000000000000000000000000000 P 677b2c4e232e4423bdf25f46ac843637 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 23:01:52.046079 27797 sys_catalog.cc:455] T 00000000000000000000000000000000 P 677b2c4e232e4423bdf25f46ac843637 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "677b2c4e232e4423bdf25f46ac843637" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "677b2c4e232e4423bdf25f46ac843637" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33721 } } }
I20260807 23:01:52.046105 27798 sys_catalog.cc:455] T 00000000000000000000000000000000 P 677b2c4e232e4423bdf25f46ac843637 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 677b2c4e232e4423bdf25f46ac843637. Latest consensus state: current_term: 1 leader_uuid: "677b2c4e232e4423bdf25f46ac843637" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "677b2c4e232e4423bdf25f46ac843637" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33721 } } }
I20260807 23:01:52.046527 27797 sys_catalog.cc:458] T 00000000000000000000000000000000 P 677b2c4e232e4423bdf25f46ac843637 [sys.catalog]: This master's current role is: LEADER
I20260807 23:01:52.046631 27798 sys_catalog.cc:458] T 00000000000000000000000000000000 P 677b2c4e232e4423bdf25f46ac843637 [sys.catalog]: This master's current role is: LEADER
I20260807 23:01:52.049013 27810 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 23:01:52.054827 27810 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 23:01:52.064102 27810 catalog_manager.cc:1347] Generated new cluster ID: 2dd69b16aa9e4d8bb65590758d319217
I20260807 23:01:52.064244 27810 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 23:01:52.080056 27810 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 23:01:52.080578 27810 catalog_manager.cc:1504] Loading token signing keys...
I20260807 23:01:52.094269 27810 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 677b2c4e232e4423bdf25f46ac843637: Generated new TSK 0
I20260807 23:01:52.095577 27810 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 23:01:52.111402 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 27684
W20260807 23:01:52.112731 27775 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:54079
W20260807 23:01:52.113677 25976 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 27684: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 23:01:52.113890 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 27684
2026-08-07T23:01:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (2287 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786143361779964-25976-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 23:01:52 dist-test-slave-z8x9 krb5kdc[27825](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 23:01:52 dist-test-slave-z8x9 krb5kdc[27825](info): set up 2 sockets
Aug 07 23:01:52 dist-test-slave-z8x9 krb5kdc[27825](info): commencing operation
krb5kdc: starting...
W20260807 23:01:53.524797 25976 mini_kdc.cc:121] Time spent starting KDC: real 1.362s user 0.004s sys 0.044s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786143361779964-25976-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786143361779964-25976-0/krb5kdc/test-user.keytab.
Aug 07 23:01:53 dist-test-slave-z8x9 krb5kdc[27825](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143713, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T23:01:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T23:01:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786143361779964-25976-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786143361779964-25976-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 23:01:53.848853 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786143361779964-25976-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786143361779964-25976-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786143361779964-25976-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43491
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786143361779964-25976-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43491
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786143361779964-25976-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786143361779964-25976-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786143361779964-25976-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 23:01:54.197304 27843 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 23:01:54.197690 27843 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 23:01:54.197907 27843 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 23:01:54.205368 27843 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 23:01:54.205543 27843 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 23:01:54.205631 27843 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 23:01:54.205726 27843 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 23:01:54.205796 27843 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 23:01:54.215968 27843 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786143361779964-25976-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786143361779964-25976-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43491
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786143361779964-25976-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43491
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786143361779964-25976-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 23:01:54.217469 27843 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 23:01:54.218837 27843 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 23:01:54.231487 27849 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 23:01:54.231529 27855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 23:01:54.231551 27853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 23:01:54.232745 27843 server_base.cc:1034] running on GCE node
I20260807 23:01:54.233795 27843 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 23:01:54.235404 27843 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 23:01:54.236694 27843 hybrid_clock.cc:648] HybridClock initialized: now 1786143714236639 us; error 42 us; skew 500 ppm
I20260807 23:01:54.237141 27843 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 23:01:54 dist-test-slave-z8x9 krb5kdc[27825](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143714, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 23:01:54.240880 27843 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 23:01:54.242616 27843 webserver.cc:460] Webserver started at http://127.0.0.1:43911/ using document root <none> and password file <none>
I20260807 23:01:54.243340 27843 fs_manager.cc:359] Metadata directory not provided
I20260807 23:01:54.243505 27843 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 23:01:54.243829 27843 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 23:01:54.247016 27843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786143361779964-25976-0/minicluster-data/master-0/data/instance:
uuid: "3bfcdec649194105b115b10af17a0530"
format_stamp: "Formatted at 2026-08-07 23:01:54 on dist-test-slave-z8x9"
server_key: "809e46feac2b1294326a259d71ea0678"
server_key_iv: "6bcde83395509c45012e93d55896488c"
server_key_version: "encryptionkey@0"
I20260807 23:01:54.247835 27843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786143361779964-25976-0/minicluster-data/master-0/wal/instance:
uuid: "3bfcdec649194105b115b10af17a0530"
format_stamp: "Formatted at 2026-08-07 23:01:54 on dist-test-slave-z8x9"
server_key: "809e46feac2b1294326a259d71ea0678"
server_key_iv: "6bcde83395509c45012e93d55896488c"
server_key_version: "encryptionkey@0"
I20260807 23:01:54.253324 27843 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260807 23:01:54.257091 27869 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 23:01:54.258598 27843 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 23:01:54.258816 27843 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786143361779964-25976-0/minicluster-data/master-0/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786143361779964-25976-0/minicluster-data/master-0/wal
uuid: "3bfcdec649194105b115b10af17a0530"
format_stamp: "Formatted at 2026-08-07 23:01:54 on dist-test-slave-z8x9"
server_key: "809e46feac2b1294326a259d71ea0678"
server_key_iv: "6bcde83395509c45012e93d55896488c"
server_key_version: "encryptionkey@0"
I20260807 23:01:54.259055 27843 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786143361779964-25976-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786143361779964-25976-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786143361779964-25976-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 23:01:54.271713 27843 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 23:01:54.276449 27843 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 23:01:54.276777 27843 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 23:01:54.297405 27843 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43491
I20260807 23:01:54.297428 27944 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43491 every 8 connection(s)
I20260807 23:01:54.299189 27843 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
I20260807 23:01:54.302498 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 27843
I20260807 23:01:54.302711 25976 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786143361779964-25976-0/minicluster-data/master-0/wal/instance
I20260807 23:01:54.303287 25976 external_mini_cluster.cc:1428] Setting key aab46cd4860138be18400fb75bc02c52
I20260807 23:01:54.303711 27945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 23:01:54.319702 27945 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bfcdec649194105b115b10af17a0530: Bootstrap starting.
Aug 07 23:01:54 dist-test-slave-z8x9 krb5kdc[27825](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143713, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260807 23:01:54.323323 27945 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3bfcdec649194105b115b10af17a0530: Neither blocks nor log segments found. Creating new log.
I20260807 23:01:54.324610 27945 log.cc:824] T 00000000000000000000000000000000 P 3bfcdec649194105b115b10af17a0530: Log is configured to *not* fsync() on all Append() calls
I20260807 23:01:54.327776 27945 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bfcdec649194105b115b10af17a0530: No bootstrap required, opened a new log
I20260807 23:01:54.330415 27955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:54.305312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46255 (local address 127.0.0.1:43491)
0807 23:01:54.306145 (+ 833us) server_negotiation.cc:206] Beginning negotiation
0807 23:01:54.306157 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 23:01:54.306197 (+ 40us) server_negotiation.cc:407] Connection header received
0807 23:01:54.308604 (+ 2407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:01:54.308628 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:01:54.309060 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:01:54.309514 (+ 454us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:01:54.311594 (+ 2080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:54.313624 (+ 2030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:54.315665 (+ 2041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:54.316451 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:54.316613 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:54.322004 (+ 5391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:01:54.322014 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:01:54.322030 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 23:01:54.322074 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:01:54.325932 (+ 3858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:01:54.326812 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:01:54.326819 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:01:54.326826 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:01:54.327026 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:01:54.327448 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:01:54.327454 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:01:54.327457 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:01:54.328041 (+ 584us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:01:54.328399 (+ 358us) server_negotiation.cc:1035] Waiting for connection context
0807 23:01:54.328833 (+ 434us) server_negotiation.cc:299] Negotiation successful
0807 23:01:54.329340 (+ 507us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":594,"thread_start_us":377,"threads_started":1}
I20260807 23:01:54.334444 25976 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 23:01:54.336187 27945 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3bfcdec649194105b115b10af17a0530 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bfcdec649194105b115b10af17a0530" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43491 } }
I20260807 23:01:54.336596 27945 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3bfcdec649194105b115b10af17a0530 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 23:01:54.336707 27945 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3bfcdec649194105b115b10af17a0530 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bfcdec649194105b115b10af17a0530, State: Initialized, Role: FOLLOWER
I20260807 23:01:54.337428 27945 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3bfcdec649194105b115b10af17a0530 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bfcdec649194105b115b10af17a0530" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43491 } }
I20260807 23:01:54.337764 27945 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3bfcdec649194105b115b10af17a0530 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 23:01:54.337927 27945 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3bfcdec649194105b115b10af17a0530 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 23:01:54.338155 27945 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3bfcdec649194105b115b10af17a0530 [term 0 FOLLOWER]: Advancing to term 1
I20260807 23:01:54.340968 27945 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3bfcdec649194105b115b10af17a0530 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bfcdec649194105b115b10af17a0530" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43491 } }
I20260807 23:01:54.341557 27945 leader_election.cc:302] T 00000000000000000000000000000000 P 3bfcdec649194105b115b10af17a0530 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3bfcdec649194105b115b10af17a0530; no voters:
I20260807 23:01:54.342202 27945 leader_election.cc:288] T 00000000000000000000000000000000 P 3bfcdec649194105b115b10af17a0530 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 23:01:54.342309 27963 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3bfcdec649194105b115b10af17a0530 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 23:01:54.344946 27945 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3bfcdec649194105b115b10af17a0530 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 23:01:54.345446 27963 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3bfcdec649194105b115b10af17a0530 [term 1 LEADER]: Becoming Leader. State: Replica: 3bfcdec649194105b115b10af17a0530, State: Running, Role: LEADER
I20260807 23:01:54.346110 27963 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3bfcdec649194105b115b10af17a0530 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bfcdec649194105b115b10af17a0530" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43491 } }
I20260807 23:01:54.349948 27966 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bfcdec649194105b115b10af17a0530 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3bfcdec649194105b115b10af17a0530" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bfcdec649194105b115b10af17a0530" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43491 } } }
I20260807 23:01:54.350204 27965 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bfcdec649194105b115b10af17a0530 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3bfcdec649194105b115b10af17a0530. Latest consensus state: current_term: 1 leader_uuid: "3bfcdec649194105b115b10af17a0530" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bfcdec649194105b115b10af17a0530" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43491 } } }
I20260807 23:01:54.351228 27966 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bfcdec649194105b115b10af17a0530 [sys.catalog]: This master's current role is: LEADER
I20260807 23:01:54.351254 27965 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bfcdec649194105b115b10af17a0530 [sys.catalog]: This master's current role is: LEADER
I20260807 23:01:54.352447 27977 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 23:01:54.360709 27955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:54.339773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52761 (local address 127.0.0.1:43491)
0807 23:01:54.339969 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0807 23:01:54.339977 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 23:01:54.340303 (+ 326us) server_negotiation.cc:407] Connection header received
0807 23:01:54.340490 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:01:54.340495 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:01:54.340568 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:01:54.340680 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:01:54.342256 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:54.343446 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:54.344787 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:54.345459 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:54.345626 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:54.348680 (+ 3054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:01:54.348692 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:01:54.348698 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 23:01:54.348741 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:01:54.356307 (+ 7566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:01:54.357702 (+ 1395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:01:54.357711 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:01:54.357717 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:01:54.357926 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:01:54.358539 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:01:54.358547 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:01:54.358553 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:01:54.359030 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:01:54.359344 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0807 23:01:54.359830 (+ 486us) server_negotiation.cc:299] Negotiation successful
0807 23:01:54.360091 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260807 23:01:54.369413 27977 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 23:01:54.378613 27977 catalog_manager.cc:1347] Generated new cluster ID: 71de71d3c41b40ea9bf93cff6f20689c
I20260807 23:01:54.378755 27977 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 23:01:54.416260 27977 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 23:01:54.417765 27977 catalog_manager.cc:1504] Loading token signing keys...
I20260807 23:01:54.430320 27977 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3bfcdec649194105b115b10af17a0530: Generated new TSK 0
I20260807 23:01:54.431244 27977 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 23:01:54.442888 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 27843
I20260807 23:01:54.444315 27937 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 23:01:54.539749 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 27843
2026-08-07T23:01:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2423 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786143361779964-25976-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 23:01:54 dist-test-slave-z8x9 krb5kdc[28000](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 23:01:54 dist-test-slave-z8x9 krb5kdc[28000](info): set up 2 sockets
Aug 07 23:01:54 dist-test-slave-z8x9 krb5kdc[28000](info): commencing operation
krb5kdc: starting...
W20260807 23:01:55.933279 25976 mini_kdc.cc:121] Time spent starting KDC: real 1.347s user 0.003s sys 0.043s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786143361779964-25976-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786143361779964-25976-0/krb5kdc/test-user.keytab.
Aug 07 23:01:56 dist-test-slave-z8x9 krb5kdc[28000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143716, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T23:01:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T23:01:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786143361779964-25976-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786143361779964-25976-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 23:01:56.285022 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786143361779964-25976-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786143361779964-25976-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786143361779964-25976-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36213
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786143361779964-25976-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:32905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36213
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786143361779964-25976-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786143361779964-25976-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786143361779964-25976-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 23:01:56.636724 28019 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 23:01:56.637022 28019 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 23:01:56.637251 28019 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 23:01:56.644570 28019 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 23:01:56.644680 28019 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 23:01:56.644815 28019 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 23:01:56.644927 28019 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 23:01:56.645007 28019 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 23:01:56.655169 28019 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:32905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786143361779964-25976-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786143361779964-25976-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36213
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786143361779964-25976-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36213
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786143361779964-25976-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 23:01:56.656538 28019 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 23:01:56.658016 28019 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 23:01:56.672752 28026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 23:01:56.672828 28027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 23:01:56.672714 28031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 23:01:56.673846 28019 server_base.cc:1034] running on GCE node
I20260807 23:01:56.674736 28019 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 23:01:56.676213 28019 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 23:01:56.677434 28019 hybrid_clock.cc:648] HybridClock initialized: now 1786143716677403 us; error 52 us; skew 500 ppm
I20260807 23:01:56.677798 28019 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 23:01:56 dist-test-slave-z8x9 krb5kdc[28000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143716, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 23:01:56.681047 28019 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 23:01:56.682591 28019 webserver.cc:460] Webserver started at http://127.0.0.1:37465/ using document root <none> and password file <none>
I20260807 23:01:56.683288 28019 fs_manager.cc:359] Metadata directory not provided
I20260807 23:01:56.683450 28019 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 23:01:56.683779 28019 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 23:01:56.686841 28019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786143361779964-25976-0/minicluster-data/master-0/data/instance:
uuid: "55d9e7d0ef214be7be9ed6a24eb920c2"
format_stamp: "Formatted at 2026-08-07 23:01:56 on dist-test-slave-z8x9"
server_key: "b61a1e75eeb933847592c8eb720e842c"
server_key_iv: "b1231ba9bcd812fc95425e4e154e9ee1"
server_key_version: "encryptionkey@0"
I20260807 23:01:56.687625 28019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786143361779964-25976-0/minicluster-data/master-0/wal/instance:
uuid: "55d9e7d0ef214be7be9ed6a24eb920c2"
format_stamp: "Formatted at 2026-08-07 23:01:56 on dist-test-slave-z8x9"
server_key: "b61a1e75eeb933847592c8eb720e842c"
server_key_iv: "b1231ba9bcd812fc95425e4e154e9ee1"
server_key_version: "encryptionkey@0"
I20260807 23:01:56.693032 28019 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260807 23:01:56.697278 28042 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 23:01:56.698632 28019 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 23:01:56.698829 28019 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786143361779964-25976-0/minicluster-data/master-0/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786143361779964-25976-0/minicluster-data/master-0/wal
uuid: "55d9e7d0ef214be7be9ed6a24eb920c2"
format_stamp: "Formatted at 2026-08-07 23:01:56 on dist-test-slave-z8x9"
server_key: "b61a1e75eeb933847592c8eb720e842c"
server_key_iv: "b1231ba9bcd812fc95425e4e154e9ee1"
server_key_version: "encryptionkey@0"
I20260807 23:01:56.699046 28019 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786143361779964-25976-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786143361779964-25976-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786143361779964-25976-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 23:01:56.724963 28019 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 23:01:56.730021 28019 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 23:01:56.730353 28019 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 23:01:56.750090 28019 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36213
I20260807 23:01:56.750140 28127 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36213 every 8 connection(s)
I20260807 23:01:56.751545 28019 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
I20260807 23:01:56.755321 28128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 23:01:56.759531 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 28019
I20260807 23:01:56.759747 25976 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786143361779964-25976-0/minicluster-data/master-0/wal/instance
I20260807 23:01:56.760339 25976 external_mini_cluster.cc:1428] Setting key 9c30345fc49319ae5fb8e2c15824ae06
I20260807 23:01:56.766803 28128 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55d9e7d0ef214be7be9ed6a24eb920c2: Bootstrap starting.
I20260807 23:01:56.770363 28128 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 55d9e7d0ef214be7be9ed6a24eb920c2: Neither blocks nor log segments found. Creating new log.
I20260807 23:01:56.771634 28128 log.cc:824] T 00000000000000000000000000000000 P 55d9e7d0ef214be7be9ed6a24eb920c2: Log is configured to *not* fsync() on all Append() calls
I20260807 23:01:56.774750 28133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:56.762171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60837 (local address 127.0.0.1:36213)
0807 23:01:56.762977 (+ 806us) server_negotiation.cc:206] Beginning negotiation
0807 23:01:56.762988 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 23:01:56.763027 (+ 39us) server_negotiation.cc:407] Connection header received
0807 23:01:56.765358 (+ 2331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:01:56.765380 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:01:56.765737 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:01:56.766032 (+ 295us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:01:56.766661 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:56.768064 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:56.770053 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:56.770803 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:56.770950 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:56.772322 (+ 1372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:01:56.772330 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:01:56.772344 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 23:01:56.772996 (+ 652us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:01:56.773105 (+ 109us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 23:01:56.773132 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:01:56.773325 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0807 23:01:56.773558 (+ 233us) server_negotiation.cc:299] Negotiation successful
0807 23:01:56.773816 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":595,"thread_start_us":391,"threads_started":1}
I20260807 23:01:56.774847 28128 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55d9e7d0ef214be7be9ed6a24eb920c2: No bootstrap required, opened a new log
I20260807 23:01:56.778517 25976 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 23:01:56.782723 28128 raft_consensus.cc:348] T 00000000000000000000000000000000 P 55d9e7d0ef214be7be9ed6a24eb920c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55d9e7d0ef214be7be9ed6a24eb920c2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36213 } }
I20260807 23:01:56.783243 28128 raft_consensus.cc:374] T 00000000000000000000000000000000 P 55d9e7d0ef214be7be9ed6a24eb920c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 23:01:56.783361 28128 raft_consensus.cc:729] T 00000000000000000000000000000000 P 55d9e7d0ef214be7be9ed6a24eb920c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55d9e7d0ef214be7be9ed6a24eb920c2, State: Initialized, Role: FOLLOWER
I20260807 23:01:56.784070 28128 consensus_queue.cc:260] T 00000000000000000000000000000000 P 55d9e7d0ef214be7be9ed6a24eb920c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55d9e7d0ef214be7be9ed6a24eb920c2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36213 } }
I20260807 23:01:56.784397 28128 raft_consensus.cc:388] T 00000000000000000000000000000000 P 55d9e7d0ef214be7be9ed6a24eb920c2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 23:01:56.784574 28128 raft_consensus.cc:482] T 00000000000000000000000000000000 P 55d9e7d0ef214be7be9ed6a24eb920c2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 23:01:56.784796 28128 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 55d9e7d0ef214be7be9ed6a24eb920c2 [term 0 FOLLOWER]: Advancing to term 1
I20260807 23:01:56.788017 28128 raft_consensus.cc:504] T 00000000000000000000000000000000 P 55d9e7d0ef214be7be9ed6a24eb920c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55d9e7d0ef214be7be9ed6a24eb920c2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36213 } }
I20260807 23:01:56.788767 28128 leader_election.cc:302] T 00000000000000000000000000000000 P 55d9e7d0ef214be7be9ed6a24eb920c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 55d9e7d0ef214be7be9ed6a24eb920c2; no voters:
I20260807 23:01:56.789500 28128 leader_election.cc:288] T 00000000000000000000000000000000 P 55d9e7d0ef214be7be9ed6a24eb920c2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 23:01:56.789649 28146 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 55d9e7d0ef214be7be9ed6a24eb920c2 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 23:01:56.790644 28133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:56.783119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36205 (local address 127.0.0.1:36213)
0807 23:01:56.783305 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0807 23:01:56.783313 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 23:01:56.783641 (+ 328us) server_negotiation.cc:407] Connection header received
0807 23:01:56.783842 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:01:56.783848 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:01:56.783929 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:01:56.784013 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:01:56.784739 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:56.785639 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:56.787514 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:01:56.787968 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:01:56.788070 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:01:56.789387 (+ 1317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:01:56.789395 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:01:56.789402 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 23:01:56.789431 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:01:56.789484 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 23:01:56.789514 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:01:56.789624 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0807 23:01:56.790206 (+ 582us) server_negotiation.cc:299] Negotiation successful
0807 23:01:56.790307 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260807 23:01:56.791687 28146 raft_consensus.cc:686] T 00000000000000000000000000000000 P 55d9e7d0ef214be7be9ed6a24eb920c2 [term 1 LEADER]: Becoming Leader. State: Replica: 55d9e7d0ef214be7be9ed6a24eb920c2, State: Running, Role: LEADER
I20260807 23:01:56.791996 28128 sys_catalog.cc:564] T 00000000000000000000000000000000 P 55d9e7d0ef214be7be9ed6a24eb920c2 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 23:01:56.792351 28146 consensus_queue.cc:237] T 00000000000000000000000000000000 P 55d9e7d0ef214be7be9ed6a24eb920c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55d9e7d0ef214be7be9ed6a24eb920c2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36213 } }
I20260807 23:01:56.800549 28148 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55d9e7d0ef214be7be9ed6a24eb920c2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 55d9e7d0ef214be7be9ed6a24eb920c2. Latest consensus state: current_term: 1 leader_uuid: "55d9e7d0ef214be7be9ed6a24eb920c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55d9e7d0ef214be7be9ed6a24eb920c2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36213 } } }
I20260807 23:01:56.801059 28148 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55d9e7d0ef214be7be9ed6a24eb920c2 [sys.catalog]: This master's current role is: LEADER
I20260807 23:01:56.801683 28147 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55d9e7d0ef214be7be9ed6a24eb920c2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "55d9e7d0ef214be7be9ed6a24eb920c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55d9e7d0ef214be7be9ed6a24eb920c2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36213 } } }
I20260807 23:01:56.802031 28147 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55d9e7d0ef214be7be9ed6a24eb920c2 [sys.catalog]: This master's current role is: LEADER
I20260807 23:01:56.802810 28162 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 23:01:56.807367 28162 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 23:01:56.815657 28162 catalog_manager.cc:1347] Generated new cluster ID: 883bd26ca8004fb4a9d8c829ae17d67c
I20260807 23:01:56.815794 28162 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 23:01:56.845547 28162 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 23:01:56.847506 28162 catalog_manager.cc:1504] Loading token signing keys...
I20260807 23:01:56.858788 28162 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 55d9e7d0ef214be7be9ed6a24eb920c2: Generated new TSK 0
I20260807 23:01:56.859640 28162 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 23:01:56.870507 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 28019
W20260807 23:01:56.871560 28123 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:60837
W20260807 23:01:56.872256 25976 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 28019: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 23:01:56.872402 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 28019
2026-08-07T23:01:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (2330 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786143361779964-25976-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 23:01:56 dist-test-slave-z8x9 krb5kdc[28175](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 23:01:56 dist-test-slave-z8x9 krb5kdc[28175](info): set up 2 sockets
Aug 07 23:01:56 dist-test-slave-z8x9 krb5kdc[28175](info): commencing operation
krb5kdc: starting...
W20260807 23:01:58.260814 25976 mini_kdc.cc:121] Time spent starting KDC: real 1.345s user 0.001s sys 0.043s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786143361779964-25976-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786143361779964-25976-0/krb5kdc/test-user.keytab.
Aug 07 23:01:58 dist-test-slave-z8x9 krb5kdc[28175](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143718, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T23:01:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T23:01:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786143361779964-25976-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786143361779964-25976-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 23:01:58.613106 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786143361779964-25976-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786143361779964-25976-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786143361779964-25976-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42681
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786143361779964-25976-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42681
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786143361779964-25976-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786143361779964-25976-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786143361779964-25976-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 23:01:58.963091 28190 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 23:01:58.963472 28190 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 23:01:58.963644 28190 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 23:01:58.971158 28190 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 23:01:58.971345 28190 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 23:01:58.971462 28190 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 23:01:58.971575 28190 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 23:01:58.971659 28190 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 23:01:58.981916 28190 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786143361779964-25976-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786143361779964-25976-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42681
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786143361779964-25976-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42681
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786143361779964-25976-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 23:01:58.983340 28190 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 23:01:58.984860 28190 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 23:01:58.997563 28199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 23:01:58.997641 28198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 23:01:58.997563 28203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 23:01:58.998493 28190 server_base.cc:1034] running on GCE node
I20260807 23:01:58.999341 28190 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 23:01:59.000780 28190 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 23:01:59.001994 28190 hybrid_clock.cc:648] HybridClock initialized: now 1786143719001954 us; error 29 us; skew 500 ppm
I20260807 23:01:59.002362 28190 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 23:01:58 dist-test-slave-z8x9 krb5kdc[28175](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143718, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 23:01:59.005764 28190 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 23:01:59.007387 28190 webserver.cc:460] Webserver started at http://127.0.0.1:34547/ using document root <none> and password file <none>
I20260807 23:01:59.008090 28190 fs_manager.cc:359] Metadata directory not provided
I20260807 23:01:59.008239 28190 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 23:01:59.008556 28190 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 23:01:59.011634 28190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786143361779964-25976-0/minicluster-data/master-0/data/instance:
uuid: "01b0b86c5dd04a54ab006f962dc1e815"
format_stamp: "Formatted at 2026-08-07 23:01:59 on dist-test-slave-z8x9"
server_key: "92c4ed224f19c17d8546eb34cf8f2b06"
server_key_iv: "b2c88cd409c7e5ae9d9e290a1b26878f"
server_key_version: "encryptionkey@0"
I20260807 23:01:59.012439 28190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786143361779964-25976-0/minicluster-data/master-0/wal/instance:
uuid: "01b0b86c5dd04a54ab006f962dc1e815"
format_stamp: "Formatted at 2026-08-07 23:01:59 on dist-test-slave-z8x9"
server_key: "92c4ed224f19c17d8546eb34cf8f2b06"
server_key_iv: "b2c88cd409c7e5ae9d9e290a1b26878f"
server_key_version: "encryptionkey@0"
I20260807 23:01:59.017984 28190 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260807 23:01:59.021481 28213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 23:01:59.023109 28190 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260807 23:01:59.023341 28190 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786143361779964-25976-0/minicluster-data/master-0/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786143361779964-25976-0/minicluster-data/master-0/wal
uuid: "01b0b86c5dd04a54ab006f962dc1e815"
format_stamp: "Formatted at 2026-08-07 23:01:59 on dist-test-slave-z8x9"
server_key: "92c4ed224f19c17d8546eb34cf8f2b06"
server_key_iv: "b2c88cd409c7e5ae9d9e290a1b26878f"
server_key_version: "encryptionkey@0"
I20260807 23:01:59.023554 28190 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786143361779964-25976-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786143361779964-25976-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786143361779964-25976-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 23:01:59.029886 28190 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 23:01:59.034441 28190 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 23:01:59.034755 28190 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 23:01:59.053988 28190 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42681
I20260807 23:01:59.054044 28295 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42681 every 8 connection(s)
I20260807 23:01:59.055428 28190 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
I20260807 23:01:59.059324 28296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 23:01:59.064730 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 28190
I20260807 23:01:59.064931 25976 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786143361779964-25976-0/minicluster-data/master-0/wal/instance
I20260807 23:01:59.065454 25976 external_mini_cluster.cc:1428] Setting key b8eec7086533eb57af6cc11ee5a5012c
I20260807 23:01:59.070907 28296 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01b0b86c5dd04a54ab006f962dc1e815: Bootstrap starting.
I20260807 23:01:59.073597 28302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:59.067346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52469 (local address 127.0.0.1:42681)
0807 23:01:59.068217 (+ 871us) server_negotiation.cc:206] Beginning negotiation
0807 23:01:59.068228 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 23:01:59.068276 (+ 48us) server_negotiation.cc:407] Connection header received
0807 23:01:59.070081 (+ 1805us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:01:59.070106 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:01:59.070450 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:01:59.070796 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:01:59.071898 (+ 1102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:01:59.071905 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:01:59.071917 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 23:01:59.071931 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:01:59.071995 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 23:01:59.072014 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:01:59.072190 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0807 23:01:59.072321 (+ 131us) server_negotiation.cc:299] Negotiation successful
0807 23:01:59.072539 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":641,"thread_start_us":414,"threads_started":1}
I20260807 23:01:59.074585 28296 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 01b0b86c5dd04a54ab006f962dc1e815: Neither blocks nor log segments found. Creating new log.
I20260807 23:01:59.075676 28296 log.cc:824] T 00000000000000000000000000000000 P 01b0b86c5dd04a54ab006f962dc1e815: Log is configured to *not* fsync() on all Append() calls
I20260807 23:01:59.077445 25976 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 23:01:59.079283 28296 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01b0b86c5dd04a54ab006f962dc1e815: No bootstrap required, opened a new log
I20260807 23:01:59.085595 28302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:01:59.082823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47047 (local address 127.0.0.1:42681)
0807 23:01:59.083041 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0807 23:01:59.083047 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 23:01:59.083375 (+ 328us) server_negotiation.cc:407] Connection header received
0807 23:01:59.083537 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:01:59.083540 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:01:59.083590 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:01:59.083678 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:01:59.084801 (+ 1123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:01:59.084805 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:01:59.084808 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 23:01:59.084826 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:01:59.084864 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 23:01:59.084883 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:01:59.084981 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0807 23:01:59.085112 (+ 131us) server_negotiation.cc:299] Negotiation successful
0807 23:01:59.085163 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260807 23:01:59.088992 28296 raft_consensus.cc:348] T 00000000000000000000000000000000 P 01b0b86c5dd04a54ab006f962dc1e815 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01b0b86c5dd04a54ab006f962dc1e815" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42681 } }
I20260807 23:01:59.089490 28296 raft_consensus.cc:374] T 00000000000000000000000000000000 P 01b0b86c5dd04a54ab006f962dc1e815 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 23:01:59.089617 28296 raft_consensus.cc:729] T 00000000000000000000000000000000 P 01b0b86c5dd04a54ab006f962dc1e815 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01b0b86c5dd04a54ab006f962dc1e815, State: Initialized, Role: FOLLOWER
I20260807 23:01:59.090193 28296 consensus_queue.cc:260] T 00000000000000000000000000000000 P 01b0b86c5dd04a54ab006f962dc1e815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01b0b86c5dd04a54ab006f962dc1e815" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42681 } }
I20260807 23:01:59.090468 28296 raft_consensus.cc:388] T 00000000000000000000000000000000 P 01b0b86c5dd04a54ab006f962dc1e815 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 23:01:59.090622 28296 raft_consensus.cc:482] T 00000000000000000000000000000000 P 01b0b86c5dd04a54ab006f962dc1e815 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 23:01:59.090808 28296 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 01b0b86c5dd04a54ab006f962dc1e815 [term 0 FOLLOWER]: Advancing to term 1
I20260807 23:01:59.093068 28296 raft_consensus.cc:504] T 00000000000000000000000000000000 P 01b0b86c5dd04a54ab006f962dc1e815 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01b0b86c5dd04a54ab006f962dc1e815" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42681 } }
I20260807 23:01:59.093711 28296 leader_election.cc:302] T 00000000000000000000000000000000 P 01b0b86c5dd04a54ab006f962dc1e815 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 01b0b86c5dd04a54ab006f962dc1e815; no voters:
I20260807 23:01:59.094354 28296 leader_election.cc:288] T 00000000000000000000000000000000 P 01b0b86c5dd04a54ab006f962dc1e815 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 23:01:59.094591 28311 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 01b0b86c5dd04a54ab006f962dc1e815 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 23:01:59.095476 28311 raft_consensus.cc:686] T 00000000000000000000000000000000 P 01b0b86c5dd04a54ab006f962dc1e815 [term 1 LEADER]: Becoming Leader. State: Replica: 01b0b86c5dd04a54ab006f962dc1e815, State: Running, Role: LEADER
I20260807 23:01:59.096050 28311 consensus_queue.cc:237] T 00000000000000000000000000000000 P 01b0b86c5dd04a54ab006f962dc1e815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01b0b86c5dd04a54ab006f962dc1e815" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42681 } }
I20260807 23:01:59.096683 28296 sys_catalog.cc:564] T 00000000000000000000000000000000 P 01b0b86c5dd04a54ab006f962dc1e815 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 23:01:59.099004 28315 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01b0b86c5dd04a54ab006f962dc1e815 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 01b0b86c5dd04a54ab006f962dc1e815. Latest consensus state: current_term: 1 leader_uuid: "01b0b86c5dd04a54ab006f962dc1e815" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01b0b86c5dd04a54ab006f962dc1e815" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42681 } } }
I20260807 23:01:59.099479 28315 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01b0b86c5dd04a54ab006f962dc1e815 [sys.catalog]: This master's current role is: LEADER
I20260807 23:01:59.100171 28312 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01b0b86c5dd04a54ab006f962dc1e815 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "01b0b86c5dd04a54ab006f962dc1e815" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01b0b86c5dd04a54ab006f962dc1e815" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42681 } } }
I20260807 23:01:59.100517 28312 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01b0b86c5dd04a54ab006f962dc1e815 [sys.catalog]: This master's current role is: LEADER
I20260807 23:01:59.101794 28318 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 23:01:59.106683 28318 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 23:01:59.117005 28318 catalog_manager.cc:1347] Generated new cluster ID: 0380828299d9438ba717a4d6149a11da
I20260807 23:01:59.117208 28318 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 23:01:59.145567 28318 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 23:01:59.146071 28318 catalog_manager.cc:1504] Loading token signing keys...
I20260807 23:01:59.155282 28318 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 01b0b86c5dd04a54ab006f962dc1e815: Generated new TSK 0
I20260807 23:01:59.156401 28318 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 23:01:59.171625 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 28190
W20260807 23:01:59.173097 28289 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:52469
W20260807 23:01:59.173913 25976 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 28190: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 23:01:59.174088 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 28190
2026-08-07T23:01:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (2306 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786143361779964-25976-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 23:01:59 dist-test-slave-z8x9 krb5kdc[28338](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 23:01:59 dist-test-slave-z8x9 krb5kdc[28338](info): set up 2 sockets
Aug 07 23:01:59 dist-test-slave-z8x9 krb5kdc[28338](info): commencing operation
krb5kdc: starting...
W20260807 23:02:00.567471 25976 mini_kdc.cc:121] Time spent starting KDC: real 1.344s user 0.006s sys 0.040s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786143361779964-25976-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786143361779964-25976-0/krb5kdc/test-user.keytab.
Aug 07 23:02:00 dist-test-slave-z8x9 krb5kdc[28338](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143720, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T23:02:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T23:02:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786143361779964-25976-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786143361779964-25976-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 23:02:00.901304 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786143361779964-25976-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786143361779964-25976-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786143361779964-25976-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39247
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786143361779964-25976-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39247
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786143361779964-25976-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786143361779964-25976-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786143361779964-25976-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 23:02:01.273049 28357 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 23:02:01.273483 28357 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 23:02:01.273591 28357 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 23:02:01.281014 28357 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 23:02:01.281140 28357 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 23:02:01.281261 28357 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 23:02:01.281385 28357 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 23:02:01.281469 28357 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 23:02:01.292006 28357 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786143361779964-25976-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786143361779964-25976-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39247
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786143361779964-25976-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39247
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786143361779964-25976-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 23:02:01.293546 28357 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 23:02:01.294902 28357 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 23:02:01.307427 28364 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 23:02:01.307474 28369 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 23:02:01.307474 28366 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 23:02:01.308072 28357 server_base.cc:1034] running on GCE node
I20260807 23:02:01.308894 28357 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 23:02:01.310598 28357 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 23:02:01.311857 28357 hybrid_clock.cc:648] HybridClock initialized: now 1786143721311812 us; error 44 us; skew 500 ppm
I20260807 23:02:01.312271 28357 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 23:02:01 dist-test-slave-z8x9 krb5kdc[28338](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143721, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 23:02:01.319629 28357 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 23:02:01.321493 28357 webserver.cc:460] Webserver started at http://127.0.0.1:39437/ using document root <none> and password file <none>
I20260807 23:02:01.322261 28357 fs_manager.cc:359] Metadata directory not provided
I20260807 23:02:01.322428 28357 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 23:02:01.322760 28357 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 23:02:01.326253 28357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786143361779964-25976-0/minicluster-data/master-0/data/instance:
uuid: "14dd19682e0d461490d4d6bf2b76ab0c"
format_stamp: "Formatted at 2026-08-07 23:02:01 on dist-test-slave-z8x9"
server_key: "019a2b67d11cc302125e14116836b555"
server_key_iv: "576d7183c6a51ae673dbd522c0a49017"
server_key_version: "encryptionkey@0"
I20260807 23:02:01.327099 28357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786143361779964-25976-0/minicluster-data/master-0/wal/instance:
uuid: "14dd19682e0d461490d4d6bf2b76ab0c"
format_stamp: "Formatted at 2026-08-07 23:02:01 on dist-test-slave-z8x9"
server_key: "019a2b67d11cc302125e14116836b555"
server_key_iv: "576d7183c6a51ae673dbd522c0a49017"
server_key_version: "encryptionkey@0"
I20260807 23:02:01.332903 28357 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260807 23:02:01.337443 28381 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 23:02:01.339176 28357 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260807 23:02:01.339429 28357 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786143361779964-25976-0/minicluster-data/master-0/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786143361779964-25976-0/minicluster-data/master-0/wal
uuid: "14dd19682e0d461490d4d6bf2b76ab0c"
format_stamp: "Formatted at 2026-08-07 23:02:01 on dist-test-slave-z8x9"
server_key: "019a2b67d11cc302125e14116836b555"
server_key_iv: "576d7183c6a51ae673dbd522c0a49017"
server_key_version: "encryptionkey@0"
I20260807 23:02:01.339681 28357 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786143361779964-25976-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786143361779964-25976-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786143361779964-25976-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 23:02:01.358510 28357 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 23:02:01.363822 28357 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 23:02:01.364178 28357 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 23:02:01.384496 28357 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39247
I20260807 23:02:01.384516 28460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39247 every 8 connection(s)
I20260807 23:02:01.386031 28357 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
I20260807 23:02:01.390057 28461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 23:02:01.396091 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 28357
I20260807 23:02:01.396307 25976 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786143361779964-25976-0/minicluster-data/master-0/wal/instance
I20260807 23:02:01.396780 25976 external_mini_cluster.cc:1428] Setting key 2bb0014dfb36e92838743e3b421c9f7f
I20260807 23:02:01.401654 28461 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14dd19682e0d461490d4d6bf2b76ab0c: Bootstrap starting.
I20260807 23:02:01.405471 28461 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 14dd19682e0d461490d4d6bf2b76ab0c: Neither blocks nor log segments found. Creating new log.
I20260807 23:02:01.406683 28461 log.cc:824] T 00000000000000000000000000000000 P 14dd19682e0d461490d4d6bf2b76ab0c: Log is configured to *not* fsync() on all Append() calls
I20260807 23:02:01.410089 28461 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14dd19682e0d461490d4d6bf2b76ab0c: No bootstrap required, opened a new log
Aug 07 23:02:01 dist-test-slave-z8x9 krb5kdc[28338](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143720, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260807 23:02:01.418476 28461 raft_consensus.cc:348] T 00000000000000000000000000000000 P 14dd19682e0d461490d4d6bf2b76ab0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14dd19682e0d461490d4d6bf2b76ab0c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39247 } }
I20260807 23:02:01.418913 28461 raft_consensus.cc:374] T 00000000000000000000000000000000 P 14dd19682e0d461490d4d6bf2b76ab0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 23:02:01.419059 28461 raft_consensus.cc:729] T 00000000000000000000000000000000 P 14dd19682e0d461490d4d6bf2b76ab0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14dd19682e0d461490d4d6bf2b76ab0c, State: Initialized, Role: FOLLOWER
I20260807 23:02:01.419641 28461 consensus_queue.cc:260] T 00000000000000000000000000000000 P 14dd19682e0d461490d4d6bf2b76ab0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14dd19682e0d461490d4d6bf2b76ab0c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39247 } }
I20260807 23:02:01.419917 28461 raft_consensus.cc:388] T 00000000000000000000000000000000 P 14dd19682e0d461490d4d6bf2b76ab0c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 23:02:01.420091 28461 raft_consensus.cc:482] T 00000000000000000000000000000000 P 14dd19682e0d461490d4d6bf2b76ab0c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 23:02:01.420291 28461 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 14dd19682e0d461490d4d6bf2b76ab0c [term 0 FOLLOWER]: Advancing to term 1
I20260807 23:02:01.422084 28465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:01.399108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.69:35745 (local address 127.0.0.1:39247)
0807 23:02:01.399974 (+ 866us) server_negotiation.cc:206] Beginning negotiation
0807 23:02:01.399986 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 23:02:01.400028 (+ 42us) server_negotiation.cc:407] Connection header received
0807 23:02:01.401876 (+ 1848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:02:01.401901 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:02:01.402317 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:02:01.402630 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:02:01.404526 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:01.406020 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:01.407787 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:01.408549 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:01.408690 (+ 141us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:02:01.413849 (+ 5159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:02:01.413858 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:02:01.413878 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 23:02:01.413923 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:02:01.417346 (+ 3423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:02:01.418193 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:02:01.418200 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:02:01.418206 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:02:01.418396 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:02:01.418915 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:02:01.418927 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:02:01.418930 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:02:01.419669 (+ 739us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:02:01.420016 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0807 23:02:01.420598 (+ 582us) server_negotiation.cc:299] Negotiation successful
0807 23:02:01.420929 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":640,"thread_start_us":423,"threads_started":1}
I20260807 23:02:01.423072 28461 raft_consensus.cc:504] T 00000000000000000000000000000000 P 14dd19682e0d461490d4d6bf2b76ab0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14dd19682e0d461490d4d6bf2b76ab0c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39247 } }
I20260807 23:02:01.423679 28461 leader_election.cc:302] T 00000000000000000000000000000000 P 14dd19682e0d461490d4d6bf2b76ab0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 14dd19682e0d461490d4d6bf2b76ab0c; no voters:
I20260807 23:02:01.424305 28461 leader_election.cc:288] T 00000000000000000000000000000000 P 14dd19682e0d461490d4d6bf2b76ab0c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 23:02:01.424521 28467 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 14dd19682e0d461490d4d6bf2b76ab0c [term 1 FOLLOWER]: Leader election won for term 1
I20260807 23:02:01.425988 28467 raft_consensus.cc:686] T 00000000000000000000000000000000 P 14dd19682e0d461490d4d6bf2b76ab0c [term 1 LEADER]: Becoming Leader. State: Replica: 14dd19682e0d461490d4d6bf2b76ab0c, State: Running, Role: LEADER
I20260807 23:02:01.426685 28467 consensus_queue.cc:237] T 00000000000000000000000000000000 P 14dd19682e0d461490d4d6bf2b76ab0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14dd19682e0d461490d4d6bf2b76ab0c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39247 } }
I20260807 23:02:01.426821 25976 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 23:02:01.427170 28461 sys_catalog.cc:564] T 00000000000000000000000000000000 P 14dd19682e0d461490d4d6bf2b76ab0c [sys.catalog]: configured and running, proceeding with master startup.
I20260807 23:02:01.429471 28468 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14dd19682e0d461490d4d6bf2b76ab0c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "14dd19682e0d461490d4d6bf2b76ab0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14dd19682e0d461490d4d6bf2b76ab0c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39247 } } }
I20260807 23:02:01.429615 28469 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14dd19682e0d461490d4d6bf2b76ab0c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 14dd19682e0d461490d4d6bf2b76ab0c. Latest consensus state: current_term: 1 leader_uuid: "14dd19682e0d461490d4d6bf2b76ab0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14dd19682e0d461490d4d6bf2b76ab0c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39247 } } }
I20260807 23:02:01.429956 28468 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14dd19682e0d461490d4d6bf2b76ab0c [sys.catalog]: This master's current role is: LEADER
I20260807 23:02:01.429956 28469 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14dd19682e0d461490d4d6bf2b76ab0c [sys.catalog]: This master's current role is: LEADER
I20260807 23:02:01.431094 28473 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 23:02:01.447136 28473 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 23:02:01.473341 28473 catalog_manager.cc:1347] Generated new cluster ID: fa87111011e14baca55968ee69d99a55
I20260807 23:02:01.473483 28473 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 23:02:01.493757 28465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:01.460024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.69:55767 (local address 127.0.0.1:39247)
0807 23:02:01.460262 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0807 23:02:01.460271 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 23:02:01.470882 (+ 10611us) server_negotiation.cc:407] Connection header received
0807 23:02:01.471591 (+ 709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:02:01.471601 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:02:01.471721 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:02:01.471816 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:02:01.474791 (+ 2975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:01.476446 (+ 1655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:01.479052 (+ 2606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:01.480008 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:01.480159 (+ 151us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:02:01.485054 (+ 4895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:02:01.485066 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:02:01.485074 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 23:02:01.485146 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:02:01.489444 (+ 4298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:02:01.490799 (+ 1355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:02:01.490819 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:02:01.490823 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:02:01.491037 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:02:01.491696 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:02:01.491701 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:02:01.491705 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:02:01.492135 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:02:01.492399 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0807 23:02:01.493121 (+ 722us) server_negotiation.cc:299] Negotiation successful
0807 23:02:01.493360 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260807 23:02:01.496203 28503 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260807 23:02:01.508301 28473 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 23:02:01.510296 28473 catalog_manager.cc:1504] Loading token signing keys...
I20260807 23:02:01.524783 28473 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 14dd19682e0d461490d4d6bf2b76ab0c: Generated new TSK 0
I20260807 23:02:01.525909 28473 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 23:02:01.539479 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 28357
I20260807 23:02:01.541533 28455 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 23:02:01.635661 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 28357
W20260807 23:02:01.660444 28335 connection.cc:535] client connection to 127.0.0.1:39247 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39247): Connection reset by peer (error 104)
2026-08-07T23:02:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2456 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786143361779964-25976-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 23:02:01 dist-test-slave-z8x9 krb5kdc[28512](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 23:02:01 dist-test-slave-z8x9 krb5kdc[28512](info): set up 2 sockets
Aug 07 23:02:01 dist-test-slave-z8x9 krb5kdc[28512](info): commencing operation
krb5kdc: starting...
W20260807 23:02:03.017277 25976 mini_kdc.cc:121] Time spent starting KDC: real 1.337s user 0.000s sys 0.045s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786143361779964-25976-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786143361779964-25976-0/krb5kdc/test-user.keytab.
Aug 07 23:02:03 dist-test-slave-z8x9 krb5kdc[28512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143723, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T23:02:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T23:02:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786143361779964-25976-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786143361779964-25976-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 23:02:03.365787 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786143361779964-25976-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786143361779964-25976-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786143361779964-25976-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42427
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786143361779964-25976-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42427
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786143361779964-25976-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786143361779964-25976-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786143361779964-25976-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 23:02:03.713670 28538 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 23:02:03.713986 28538 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 23:02:03.714151 28538 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 23:02:03.721470 28538 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 23:02:03.721585 28538 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 23:02:03.721681 28538 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 23:02:03.721774 28538 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 23:02:03.721853 28538 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 23:02:03.731599 28538 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786143361779964-25976-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786143361779964-25976-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42427
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786143361779964-25976-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42427
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786143361779964-25976-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 23:02:03.732951 28538 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 23:02:03.734288 28538 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 23:02:03.746146 28545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 23:02:03.746223 28544 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 23:02:03.746189 28548 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 23:02:03.747035 28538 server_base.cc:1034] running on GCE node
I20260807 23:02:03.747912 28538 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 23:02:03.749460 28538 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 23:02:03.750682 28538 hybrid_clock.cc:648] HybridClock initialized: now 1786143723750651 us; error 41 us; skew 500 ppm
I20260807 23:02:03.751111 28538 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 23:02:03 dist-test-slave-z8x9 krb5kdc[28512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143723, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 23:02:03.754559 28538 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 23:02:03.756078 28538 webserver.cc:460] Webserver started at http://127.0.0.1:42593/ using document root <none> and password file <none>
I20260807 23:02:03.756790 28538 fs_manager.cc:359] Metadata directory not provided
I20260807 23:02:03.756947 28538 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 23:02:03.757336 28538 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 23:02:03.760450 28538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786143361779964-25976-0/minicluster-data/master-0/data/instance:
uuid: "a1297b68bef44512b9e6a994896697a6"
format_stamp: "Formatted at 2026-08-07 23:02:03 on dist-test-slave-z8x9"
server_key: "b380fec352b24c8f79353548e21bb6cb"
server_key_iv: "8dcb8484ddd9a53bdbbc328710c2e463"
server_key_version: "encryptionkey@0"
I20260807 23:02:03.761328 28538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786143361779964-25976-0/minicluster-data/master-0/wal/instance:
uuid: "a1297b68bef44512b9e6a994896697a6"
format_stamp: "Formatted at 2026-08-07 23:02:03 on dist-test-slave-z8x9"
server_key: "b380fec352b24c8f79353548e21bb6cb"
server_key_iv: "8dcb8484ddd9a53bdbbc328710c2e463"
server_key_version: "encryptionkey@0"
I20260807 23:02:03.766750 28538 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260807 23:02:03.770237 28559 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 23:02:03.771510 28538 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260807 23:02:03.771700 28538 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786143361779964-25976-0/minicluster-data/master-0/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786143361779964-25976-0/minicluster-data/master-0/wal
uuid: "a1297b68bef44512b9e6a994896697a6"
format_stamp: "Formatted at 2026-08-07 23:02:03 on dist-test-slave-z8x9"
server_key: "b380fec352b24c8f79353548e21bb6cb"
server_key_iv: "8dcb8484ddd9a53bdbbc328710c2e463"
server_key_version: "encryptionkey@0"
I20260807 23:02:03.771917 28538 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786143361779964-25976-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786143361779964-25976-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786143361779964-25976-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 23:02:03.799063 28538 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 23:02:03.803400 28538 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 23:02:03.803701 28538 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 23:02:03.823009 28538 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42427
I20260807 23:02:03.823033 28638 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42427 every 8 connection(s)
I20260807 23:02:03.824406 28538 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
I20260807 23:02:03.828056 28639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 23:02:03.829667 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 28538
I20260807 23:02:03.829857 25976 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786143361779964-25976-0/minicluster-data/master-0/wal/instance
I20260807 23:02:03.830359 25976 external_mini_cluster.cc:1428] Setting key 99aad4e9789866a5531f1f62c8319ce1
I20260807 23:02:03.841997 28639 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1297b68bef44512b9e6a994896697a6: Bootstrap starting.
I20260807 23:02:03.844269 28645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:03.832281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.69:58373 (local address 127.0.0.1:42427)
0807 23:02:03.833157 (+ 876us) server_negotiation.cc:206] Beginning negotiation
0807 23:02:03.833217 (+ 60us) server_negotiation.cc:399] Waiting for connection header
0807 23:02:03.833265 (+ 48us) server_negotiation.cc:407] Connection header received
0807 23:02:03.834815 (+ 1550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:02:03.834839 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:02:03.835226 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:02:03.835527 (+ 301us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:02:03.836161 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:03.837543 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:03.839503 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:03.840169 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:03.840312 (+ 143us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:02:03.841792 (+ 1480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:02:03.841800 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:02:03.841815 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 23:02:03.842453 (+ 638us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:02:03.842543 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 23:02:03.842569 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:02:03.842700 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0807 23:02:03.843083 (+ 383us) server_negotiation.cc:299] Negotiation successful
0807 23:02:03.843277 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":630,"thread_start_us":399,"threads_started":1}
I20260807 23:02:03.845393 28639 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a1297b68bef44512b9e6a994896697a6: Neither blocks nor log segments found. Creating new log.
I20260807 23:02:03.846513 28639 log.cc:824] T 00000000000000000000000000000000 P a1297b68bef44512b9e6a994896697a6: Log is configured to *not* fsync() on all Append() calls
I20260807 23:02:03.848394 25976 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 23:02:03.849900 28639 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1297b68bef44512b9e6a994896697a6: No bootstrap required, opened a new log
I20260807 23:02:03.860702 28645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:03.853819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.69:57267 (local address 127.0.0.1:42427)
0807 23:02:03.854005 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0807 23:02:03.854011 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 23:02:03.854366 (+ 355us) server_negotiation.cc:407] Connection header received
0807 23:02:03.854528 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:02:03.854533 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:02:03.854621 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:02:03.854728 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:02:03.855218 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:03.856434 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:03.857897 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:03.858616 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:03.858754 (+ 138us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:02:03.859866 (+ 1112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:02:03.859872 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:02:03.859876 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 23:02:03.859902 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:02:03.859970 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 23:02:03.859992 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:02:03.860111 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0807 23:02:03.860270 (+ 159us) server_negotiation.cc:299] Negotiation successful
0807 23:02:03.860323 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260807 23:02:03.860430 28639 raft_consensus.cc:348] T 00000000000000000000000000000000 P a1297b68bef44512b9e6a994896697a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1297b68bef44512b9e6a994896697a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42427 } }
I20260807 23:02:03.860950 28639 raft_consensus.cc:374] T 00000000000000000000000000000000 P a1297b68bef44512b9e6a994896697a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 23:02:03.861091 28639 raft_consensus.cc:729] T 00000000000000000000000000000000 P a1297b68bef44512b9e6a994896697a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1297b68bef44512b9e6a994896697a6, State: Initialized, Role: FOLLOWER
I20260807 23:02:03.861724 28639 consensus_queue.cc:260] T 00000000000000000000000000000000 P a1297b68bef44512b9e6a994896697a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1297b68bef44512b9e6a994896697a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42427 } }
I20260807 23:02:03.861990 28639 raft_consensus.cc:388] T 00000000000000000000000000000000 P a1297b68bef44512b9e6a994896697a6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 23:02:03.862155 28639 raft_consensus.cc:482] T 00000000000000000000000000000000 P a1297b68bef44512b9e6a994896697a6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 23:02:03.862358 28639 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a1297b68bef44512b9e6a994896697a6 [term 0 FOLLOWER]: Advancing to term 1
I20260807 23:02:03.864586 28639 raft_consensus.cc:504] T 00000000000000000000000000000000 P a1297b68bef44512b9e6a994896697a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1297b68bef44512b9e6a994896697a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42427 } }
I20260807 23:02:03.865130 28639 leader_election.cc:302] T 00000000000000000000000000000000 P a1297b68bef44512b9e6a994896697a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a1297b68bef44512b9e6a994896697a6; no voters:
I20260807 23:02:03.865855 28639 leader_election.cc:288] T 00000000000000000000000000000000 P a1297b68bef44512b9e6a994896697a6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 23:02:03.866020 28662 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a1297b68bef44512b9e6a994896697a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 23:02:03.868296 28639 sys_catalog.cc:564] T 00000000000000000000000000000000 P a1297b68bef44512b9e6a994896697a6 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 23:02:03.869762 28662 raft_consensus.cc:686] T 00000000000000000000000000000000 P a1297b68bef44512b9e6a994896697a6 [term 1 LEADER]: Becoming Leader. State: Replica: a1297b68bef44512b9e6a994896697a6, State: Running, Role: LEADER
I20260807 23:02:03.870523 28662 consensus_queue.cc:237] T 00000000000000000000000000000000 P a1297b68bef44512b9e6a994896697a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1297b68bef44512b9e6a994896697a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42427 } }
I20260807 23:02:03.874032 28666 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1297b68bef44512b9e6a994896697a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a1297b68bef44512b9e6a994896697a6. Latest consensus state: current_term: 1 leader_uuid: "a1297b68bef44512b9e6a994896697a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1297b68bef44512b9e6a994896697a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42427 } } }
I20260807 23:02:03.874094 28663 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1297b68bef44512b9e6a994896697a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a1297b68bef44512b9e6a994896697a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1297b68bef44512b9e6a994896697a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42427 } } }
I20260807 23:02:03.874456 28666 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1297b68bef44512b9e6a994896697a6 [sys.catalog]: This master's current role is: LEADER
I20260807 23:02:03.874475 28663 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1297b68bef44512b9e6a994896697a6 [sys.catalog]: This master's current role is: LEADER
I20260807 23:02:03.876420 28683 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 23:02:03.881939 28683 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 23:02:03.891203 28683 catalog_manager.cc:1347] Generated new cluster ID: 1740bd2fa7d7419d88c952b41e6836a9
I20260807 23:02:03.891330 28683 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 23:02:03.924958 28683 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 23:02:03.926285 28683 catalog_manager.cc:1504] Loading token signing keys...
I20260807 23:02:03.934500 28683 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a1297b68bef44512b9e6a994896697a6: Generated new TSK 0
I20260807 23:02:03.935348 28683 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 23:02:03.950665 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 28538
W20260807 23:02:03.951916 28632 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.69:58373
W20260807 23:02:03.952713 25976 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 28538: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 23:02:03.952876 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 28538
W20260807 23:02:03.976966 28506 connection.cc:535] client connection to 127.0.0.1:42427 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42427): Connection reset by peer (error 104)
2026-08-07T23:02:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (2316 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786143361779964-25976-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 23:02:04 dist-test-slave-z8x9 krb5kdc[28693](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 23:02:04 dist-test-slave-z8x9 krb5kdc[28693](info): set up 2 sockets
Aug 07 23:02:04 dist-test-slave-z8x9 krb5kdc[28693](info): commencing operation
krb5kdc: starting...
W20260807 23:02:05.330704 25976 mini_kdc.cc:121] Time spent starting KDC: real 1.334s user 0.000s sys 0.045s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786143361779964-25976-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786143361779964-25976-0/krb5kdc/test-user.keytab.
Aug 07 23:02:05 dist-test-slave-z8x9 krb5kdc[28693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143725, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T23:02:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T23:02:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786143361779964-25976-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786143361779964-25976-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 23:02:05.662879 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786143361779964-25976-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786143361779964-25976-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786143361779964-25976-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45037
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786143361779964-25976-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45037
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786143361779964-25976-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786143361779964-25976-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786143361779964-25976-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 23:02:06.015297 28721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 23:02:06.015585 28721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 23:02:06.015761 28721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 23:02:06.023057 28721 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 23:02:06.023211 28721 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 23:02:06.023315 28721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 23:02:06.023414 28721 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 23:02:06.023479 28721 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 23:02:06.033556 28721 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786143361779964-25976-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786143361779964-25976-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45037
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786143361779964-25976-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45037
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786143361779964-25976-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 23:02:06.034876 28721 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 23:02:06.036232 28721 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 23:02:06.048826 28734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 23:02:06.048898 28729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 23:02:06.048808 28728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 23:02:06.050134 28721 server_base.cc:1034] running on GCE node
I20260807 23:02:06.051172 28721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 23:02:06.052896 28721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 23:02:06.054148 28721 hybrid_clock.cc:648] HybridClock initialized: now 1786143726054084 us; error 82 us; skew 500 ppm
I20260807 23:02:06.054554 28721 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 23:02:06 dist-test-slave-z8x9 krb5kdc[28693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143726, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 23:02:06.058225 28721 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 23:02:06.059742 28721 webserver.cc:460] Webserver started at http://127.0.0.1:44065/ using document root <none> and password file <none>
I20260807 23:02:06.060410 28721 fs_manager.cc:359] Metadata directory not provided
I20260807 23:02:06.060516 28721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 23:02:06.060778 28721 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 23:02:06.063936 28721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786143361779964-25976-0/minicluster-data/master-0/data/instance:
uuid: "a713250bcf2e463d99d28e2a9bd597d4"
format_stamp: "Formatted at 2026-08-07 23:02:06 on dist-test-slave-z8x9"
server_key: "d884aaee4fdad36e79fa6e5f1f5890dd"
server_key_iv: "8311ec88fce5af863d3faced87708c84"
server_key_version: "encryptionkey@0"
I20260807 23:02:06.064730 28721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786143361779964-25976-0/minicluster-data/master-0/wal/instance:
uuid: "a713250bcf2e463d99d28e2a9bd597d4"
format_stamp: "Formatted at 2026-08-07 23:02:06 on dist-test-slave-z8x9"
server_key: "d884aaee4fdad36e79fa6e5f1f5890dd"
server_key_iv: "8311ec88fce5af863d3faced87708c84"
server_key_version: "encryptionkey@0"
I20260807 23:02:06.070395 28721 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260807 23:02:06.074007 28742 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 23:02:06.075343 28721 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 23:02:06.075531 28721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786143361779964-25976-0/minicluster-data/master-0/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786143361779964-25976-0/minicluster-data/master-0/wal
uuid: "a713250bcf2e463d99d28e2a9bd597d4"
format_stamp: "Formatted at 2026-08-07 23:02:06 on dist-test-slave-z8x9"
server_key: "d884aaee4fdad36e79fa6e5f1f5890dd"
server_key_iv: "8311ec88fce5af863d3faced87708c84"
server_key_version: "encryptionkey@0"
I20260807 23:02:06.075740 28721 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786143361779964-25976-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786143361779964-25976-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786143361779964-25976-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 23:02:06.081996 28721 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 23:02:06.088308 28721 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 23:02:06.088698 28721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 23:02:06.115803 28721 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45037
I20260807 23:02:06.115854 28828 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45037 every 8 connection(s)
I20260807 23:02:06.117664 28721 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
I20260807 23:02:06.122025 28829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 23:02:06.125284 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 28721
I20260807 23:02:06.125502 25976 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786143361779964-25976-0/minicluster-data/master-0/wal/instance
I20260807 23:02:06.126077 25976 external_mini_cluster.cc:1428] Setting key f2ae80c465f0f94453d044753572baf7
I20260807 23:02:06.136289 28834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:06.128980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.69:50463 (local address 127.0.0.1:45037)
0807 23:02:06.130365 (+ 1385us) server_negotiation.cc:206] Beginning negotiation
0807 23:02:06.130387 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0807 23:02:06.130456 (+ 69us) server_negotiation.cc:407] Connection header received
0807 23:02:06.132322 (+ 1866us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:02:06.132350 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:02:06.132780 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:02:06.133166 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:02:06.134492 (+ 1326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:02:06.134500 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:02:06.134514 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 23:02:06.134530 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:02:06.134604 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 23:02:06.134626 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:02:06.134832 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0807 23:02:06.134938 (+ 106us) server_negotiation.cc:299] Negotiation successful
0807 23:02:06.135156 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":860,"thread_start_us":472,"threads_started":1}
I20260807 23:02:06.136842 28829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a713250bcf2e463d99d28e2a9bd597d4: Bootstrap starting.
I20260807 23:02:06.140592 25976 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 23:02:06.140599 28829 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a713250bcf2e463d99d28e2a9bd597d4: Neither blocks nor log segments found. Creating new log.
I20260807 23:02:06.141871 28829 log.cc:824] T 00000000000000000000000000000000 P a713250bcf2e463d99d28e2a9bd597d4: Log is configured to *not* fsync() on all Append() calls
I20260807 23:02:06.145574 28829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a713250bcf2e463d99d28e2a9bd597d4: No bootstrap required, opened a new log
I20260807 23:02:06.148447 28834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:06.145573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.69:40335 (local address 127.0.0.1:45037)
0807 23:02:06.145754 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0807 23:02:06.145762 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 23:02:06.146276 (+ 514us) server_negotiation.cc:407] Connection header received
0807 23:02:06.146511 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:02:06.146515 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:02:06.146566 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:02:06.146675 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:02:06.147739 (+ 1064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:02:06.147744 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:02:06.147747 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 23:02:06.147762 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:02:06.147808 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 23:02:06.147831 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:02:06.147986 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0807 23:02:06.148068 (+ 82us) server_negotiation.cc:299] Negotiation successful
0807 23:02:06.148138 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260807 23:02:06.154656 28829 raft_consensus.cc:348] T 00000000000000000000000000000000 P a713250bcf2e463d99d28e2a9bd597d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a713250bcf2e463d99d28e2a9bd597d4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45037 } }
I20260807 23:02:06.155154 28829 raft_consensus.cc:374] T 00000000000000000000000000000000 P a713250bcf2e463d99d28e2a9bd597d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 23:02:06.155267 28829 raft_consensus.cc:729] T 00000000000000000000000000000000 P a713250bcf2e463d99d28e2a9bd597d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a713250bcf2e463d99d28e2a9bd597d4, State: Initialized, Role: FOLLOWER
I20260807 23:02:06.155925 28829 consensus_queue.cc:260] T 00000000000000000000000000000000 P a713250bcf2e463d99d28e2a9bd597d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a713250bcf2e463d99d28e2a9bd597d4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45037 } }
I20260807 23:02:06.156225 28829 raft_consensus.cc:388] T 00000000000000000000000000000000 P a713250bcf2e463d99d28e2a9bd597d4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 23:02:06.156404 28829 raft_consensus.cc:482] T 00000000000000000000000000000000 P a713250bcf2e463d99d28e2a9bd597d4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 23:02:06.156610 28829 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a713250bcf2e463d99d28e2a9bd597d4 [term 0 FOLLOWER]: Advancing to term 1
I20260807 23:02:06.159122 28829 raft_consensus.cc:504] T 00000000000000000000000000000000 P a713250bcf2e463d99d28e2a9bd597d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a713250bcf2e463d99d28e2a9bd597d4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45037 } }
I20260807 23:02:06.159703 28829 leader_election.cc:302] T 00000000000000000000000000000000 P a713250bcf2e463d99d28e2a9bd597d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a713250bcf2e463d99d28e2a9bd597d4; no voters:
I20260807 23:02:06.160357 28829 leader_election.cc:288] T 00000000000000000000000000000000 P a713250bcf2e463d99d28e2a9bd597d4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 23:02:06.160614 28844 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a713250bcf2e463d99d28e2a9bd597d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 23:02:06.162360 28844 raft_consensus.cc:686] T 00000000000000000000000000000000 P a713250bcf2e463d99d28e2a9bd597d4 [term 1 LEADER]: Becoming Leader. State: Replica: a713250bcf2e463d99d28e2a9bd597d4, State: Running, Role: LEADER
I20260807 23:02:06.163070 28829 sys_catalog.cc:564] T 00000000000000000000000000000000 P a713250bcf2e463d99d28e2a9bd597d4 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 23:02:06.163111 28844 consensus_queue.cc:237] T 00000000000000000000000000000000 P a713250bcf2e463d99d28e2a9bd597d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a713250bcf2e463d99d28e2a9bd597d4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45037 } }
I20260807 23:02:06.167680 28847 sys_catalog.cc:455] T 00000000000000000000000000000000 P a713250bcf2e463d99d28e2a9bd597d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a713250bcf2e463d99d28e2a9bd597d4. Latest consensus state: current_term: 1 leader_uuid: "a713250bcf2e463d99d28e2a9bd597d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a713250bcf2e463d99d28e2a9bd597d4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45037 } } }
I20260807 23:02:06.168187 28847 sys_catalog.cc:458] T 00000000000000000000000000000000 P a713250bcf2e463d99d28e2a9bd597d4 [sys.catalog]: This master's current role is: LEADER
I20260807 23:02:06.168788 28845 sys_catalog.cc:455] T 00000000000000000000000000000000 P a713250bcf2e463d99d28e2a9bd597d4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a713250bcf2e463d99d28e2a9bd597d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a713250bcf2e463d99d28e2a9bd597d4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45037 } } }
I20260807 23:02:06.169149 28845 sys_catalog.cc:458] T 00000000000000000000000000000000 P a713250bcf2e463d99d28e2a9bd597d4 [sys.catalog]: This master's current role is: LEADER
I20260807 23:02:06.169395 28855 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 23:02:06.177685 28855 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 23:02:06.189805 28855 catalog_manager.cc:1347] Generated new cluster ID: 423855f1f7dc4f539c8bbca9a9d6c3b4
I20260807 23:02:06.190003 28855 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 23:02:06.218286 28855 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 23:02:06.218864 28855 catalog_manager.cc:1504] Loading token signing keys...
I20260807 23:02:06.230319 28855 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a713250bcf2e463d99d28e2a9bd597d4: Generated new TSK 0
I20260807 23:02:06.231447 28855 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 23:02:06.238744 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 28721
W20260807 23:02:06.239948 28824 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.69:50463
W20260807 23:02:06.240710 25976 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 28721: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 23:02:06.240877 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 28721
2026-08-07T23:02:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (2289 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786143361779964-25976-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 23:02:06 dist-test-slave-z8x9 krb5kdc[28881](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 23:02:06 dist-test-slave-z8x9 krb5kdc[28881](info): set up 2 sockets
Aug 07 23:02:06 dist-test-slave-z8x9 krb5kdc[28881](info): commencing operation
krb5kdc: starting...
W20260807 23:02:07.632438 25976 mini_kdc.cc:121] Time spent starting KDC: real 1.346s user 0.001s sys 0.046s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786143361779964-25976-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786143361779964-25976-0/krb5kdc/test-user.keytab.
Aug 07 23:02:07 dist-test-slave-z8x9 krb5kdc[28881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143727, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T23:02:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T23:02:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786143361779964-25976-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786143361779964-25976-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 23:02:07.968986 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786143361779964-25976-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786143361779964-25976-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786143361779964-25976-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44941
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786143361779964-25976-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44941
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786143361779964-25976-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786143361779964-25976-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786143361779964-25976-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 23:02:08.333750 28908 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 23:02:08.334124 28908 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 23:02:08.334312 28908 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 23:02:08.341813 28908 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 23:02:08.341981 28908 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 23:02:08.342087 28908 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 23:02:08.342186 28908 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 23:02:08.342272 28908 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 23:02:08.352843 28908 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786143361779964-25976-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786143361779964-25976-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44941
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786143361779964-25976-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44941
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786143361779964-25976-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 23:02:08.354319 28908 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 23:02:08.355708 28908 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 23:02:08.367985 28920 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 23:02:08.368014 28921 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 23:02:08.368049 28923 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 23:02:08.368902 28908 server_base.cc:1034] running on GCE node
I20260807 23:02:08.369853 28908 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 23:02:08.371383 28908 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 23:02:08.372633 28908 hybrid_clock.cc:648] HybridClock initialized: now 1786143728372568 us; error 41 us; skew 500 ppm
I20260807 23:02:08.373008 28908 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 23:02:08 dist-test-slave-z8x9 krb5kdc[28881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143728, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 23:02:08.376298 28908 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 23:02:08.377929 28908 webserver.cc:460] Webserver started at http://127.0.0.1:37209/ using document root <none> and password file <none>
I20260807 23:02:08.378643 28908 fs_manager.cc:359] Metadata directory not provided
I20260807 23:02:08.378804 28908 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 23:02:08.379143 28908 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 23:02:08.382385 28908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786143361779964-25976-0/minicluster-data/master-0/data/instance:
uuid: "11494610ac094096915141fcd4506428"
format_stamp: "Formatted at 2026-08-07 23:02:08 on dist-test-slave-z8x9"
server_key: "7be8c82f017a986372975b81797f919b"
server_key_iv: "d2664c578e8d5df62dd5511641b45bdd"
server_key_version: "encryptionkey@0"
I20260807 23:02:08.383227 28908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786143361779964-25976-0/minicluster-data/master-0/wal/instance:
uuid: "11494610ac094096915141fcd4506428"
format_stamp: "Formatted at 2026-08-07 23:02:08 on dist-test-slave-z8x9"
server_key: "7be8c82f017a986372975b81797f919b"
server_key_iv: "d2664c578e8d5df62dd5511641b45bdd"
server_key_version: "encryptionkey@0"
I20260807 23:02:08.388628 28908 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260807 23:02:08.392493 28931 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 23:02:08.394023 28908 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 23:02:08.394251 28908 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786143361779964-25976-0/minicluster-data/master-0/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786143361779964-25976-0/minicluster-data/master-0/wal
uuid: "11494610ac094096915141fcd4506428"
format_stamp: "Formatted at 2026-08-07 23:02:08 on dist-test-slave-z8x9"
server_key: "7be8c82f017a986372975b81797f919b"
server_key_iv: "d2664c578e8d5df62dd5511641b45bdd"
server_key_version: "encryptionkey@0"
I20260807 23:02:08.394476 28908 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786143361779964-25976-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786143361779964-25976-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786143361779964-25976-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 23:02:08.436349 28908 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 23:02:08.441111 28908 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 23:02:08.441469 28908 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 23:02:08.460538 28908 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44941
I20260807 23:02:08.460569 29004 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44941 every 8 connection(s)
I20260807 23:02:08.462013 28908 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
I20260807 23:02:08.463689 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 28908
I20260807 23:02:08.463901 25976 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786143361779964-25976-0/minicluster-data/master-0/wal/instance
I20260807 23:02:08.464386 25976 external_mini_cluster.cc:1428] Setting key 51c2e2052b50b24958bd71ab5355bbb1
I20260807 23:02:08.466609 29007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 07 23:02:08 dist-test-slave-z8x9 krb5kdc[28881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143727, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260807 23:02:08.480686 29007 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11494610ac094096915141fcd4506428: Bootstrap starting.
I20260807 23:02:08.484064 29007 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 11494610ac094096915141fcd4506428: Neither blocks nor log segments found. Creating new log.
I20260807 23:02:08.485284 29007 log.cc:824] T 00000000000000000000000000000000 P 11494610ac094096915141fcd4506428: Log is configured to *not* fsync() on all Append() calls
I20260807 23:02:08.488448 29007 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11494610ac094096915141fcd4506428: No bootstrap required, opened a new log
I20260807 23:02:08.489409 29010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:08.466404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.69:45625 (local address 127.0.0.1:44941)
0807 23:02:08.467342 (+ 938us) server_negotiation.cc:206] Beginning negotiation
0807 23:02:08.467355 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0807 23:02:08.467396 (+ 41us) server_negotiation.cc:407] Connection header received
0807 23:02:08.469053 (+ 1657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:02:08.469074 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:02:08.469566 (+ 492us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:02:08.469923 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:02:08.471887 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:08.473385 (+ 1498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:08.475169 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:08.475865 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:08.475998 (+ 133us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:02:08.481370 (+ 5372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:02:08.481378 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:02:08.481394 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 23:02:08.481435 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:02:08.484718 (+ 3283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:02:08.485722 (+ 1004us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:02:08.485729 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:02:08.485735 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:02:08.485922 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:02:08.486450 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:02:08.486454 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:02:08.486456 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:02:08.487037 (+ 581us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:02:08.487376 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0807 23:02:08.487937 (+ 561us) server_negotiation.cc:299] Negotiation successful
0807 23:02:08.488286 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":727,"thread_start_us":471,"threads_started":1}
I20260807 23:02:08.493932 25976 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 23:02:08.497541 29007 raft_consensus.cc:348] T 00000000000000000000000000000000 P 11494610ac094096915141fcd4506428 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11494610ac094096915141fcd4506428" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44941 } }
I20260807 23:02:08.497977 29007 raft_consensus.cc:374] T 00000000000000000000000000000000 P 11494610ac094096915141fcd4506428 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 23:02:08.498092 29007 raft_consensus.cc:729] T 00000000000000000000000000000000 P 11494610ac094096915141fcd4506428 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11494610ac094096915141fcd4506428, State: Initialized, Role: FOLLOWER
I20260807 23:02:08.498850 29007 consensus_queue.cc:260] T 00000000000000000000000000000000 P 11494610ac094096915141fcd4506428 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11494610ac094096915141fcd4506428" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44941 } }
I20260807 23:02:08.499161 29007 raft_consensus.cc:388] T 00000000000000000000000000000000 P 11494610ac094096915141fcd4506428 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 23:02:08.499348 29007 raft_consensus.cc:482] T 00000000000000000000000000000000 P 11494610ac094096915141fcd4506428 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 23:02:08.499569 29007 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 11494610ac094096915141fcd4506428 [term 0 FOLLOWER]: Advancing to term 1
I20260807 23:02:08.502755 29007 raft_consensus.cc:504] T 00000000000000000000000000000000 P 11494610ac094096915141fcd4506428 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11494610ac094096915141fcd4506428" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44941 } }
I20260807 23:02:08.503417 29007 leader_election.cc:302] T 00000000000000000000000000000000 P 11494610ac094096915141fcd4506428 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 11494610ac094096915141fcd4506428; no voters:
I20260807 23:02:08.504086 29007 leader_election.cc:288] T 00000000000000000000000000000000 P 11494610ac094096915141fcd4506428 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 23:02:08.504195 29020 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 11494610ac094096915141fcd4506428 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 23:02:08.506563 29007 sys_catalog.cc:564] T 00000000000000000000000000000000 P 11494610ac094096915141fcd4506428 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 23:02:08.510879 29020 raft_consensus.cc:686] T 00000000000000000000000000000000 P 11494610ac094096915141fcd4506428 [term 1 LEADER]: Becoming Leader. State: Replica: 11494610ac094096915141fcd4506428, State: Running, Role: LEADER
I20260807 23:02:08.511562 29020 consensus_queue.cc:237] T 00000000000000000000000000000000 P 11494610ac094096915141fcd4506428 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11494610ac094096915141fcd4506428" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44941 } }
I20260807 23:02:08.514830 29021 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11494610ac094096915141fcd4506428 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "11494610ac094096915141fcd4506428" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11494610ac094096915141fcd4506428" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44941 } } }
I20260807 23:02:08.515275 29021 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11494610ac094096915141fcd4506428 [sys.catalog]: This master's current role is: LEADER
I20260807 23:02:08.515818 29022 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11494610ac094096915141fcd4506428 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 11494610ac094096915141fcd4506428. Latest consensus state: current_term: 1 leader_uuid: "11494610ac094096915141fcd4506428" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11494610ac094096915141fcd4506428" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44941 } } }
I20260807 23:02:08.516142 29022 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11494610ac094096915141fcd4506428 [sys.catalog]: This master's current role is: LEADER
I20260807 23:02:08.519207 29010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:08.498396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.69:42905 (local address 127.0.0.1:44941)
0807 23:02:08.498621 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0807 23:02:08.498630 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 23:02:08.499215 (+ 585us) server_negotiation.cc:407] Connection header received
0807 23:02:08.499478 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:02:08.499487 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:02:08.499584 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:02:08.499697 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:02:08.501630 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:08.502400 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:08.504347 (+ 1947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:08.505313 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:08.505497 (+ 184us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:02:08.508956 (+ 3459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:02:08.508962 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:02:08.508969 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 23:02:08.509024 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:02:08.513579 (+ 4555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:02:08.514839 (+ 1260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:02:08.514845 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:02:08.514852 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:02:08.515048 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:02:08.515786 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:02:08.515794 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:02:08.515797 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:02:08.516236 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:02:08.518161 (+ 1925us) server_negotiation.cc:1035] Waiting for connection context
0807 23:02:08.518400 (+ 239us) server_negotiation.cc:299] Negotiation successful
0807 23:02:08.518646 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260807 23:02:08.526715 29040 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 23:02:08.532711 29040 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 23:02:08.542529 29040 catalog_manager.cc:1347] Generated new cluster ID: 413c662b2d4547338b0d14efdea4fc16
I20260807 23:02:08.542675 29040 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 23:02:08.603793 29040 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 23:02:08.605686 29040 catalog_manager.cc:1504] Loading token signing keys...
I20260807 23:02:08.613974 29040 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 11494610ac094096915141fcd4506428: Generated new TSK 0
I20260807 23:02:08.615190 29040 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 23:02:08.623211 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 28908
I20260807 23:02:08.624986 28995 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 23:02:08.736873 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 28908
W20260807 23:02:08.762255 28878 connection.cc:535] client connection to 127.0.0.1:44941 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44941): Connection reset by peer (error 104)
2026-08-07T23:02:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2495 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786143361779964-25976-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 23:02:08 dist-test-slave-z8x9 krb5kdc[29058](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 23:02:08 dist-test-slave-z8x9 krb5kdc[29058](info): set up 2 sockets
Aug 07 23:02:08 dist-test-slave-z8x9 krb5kdc[29058](info): commencing operation
krb5kdc: starting...
W20260807 23:02:10.112376 25976 mini_kdc.cc:121] Time spent starting KDC: real 1.330s user 0.001s sys 0.043s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786143361779964-25976-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786143361779964-25976-0/krb5kdc/test-user.keytab.
Aug 07 23:02:10 dist-test-slave-z8x9 krb5kdc[29058](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143730, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T23:02:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T23:02:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786143361779964-25976-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786143361779964-25976-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 23:02:10.464962 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786143361779964-25976-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786143361779964-25976-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786143361779964-25976-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43997
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786143361779964-25976-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43997
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786143361779964-25976-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786143361779964-25976-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786143361779964-25976-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 23:02:10.815788 29083 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 23:02:10.816097 29083 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 23:02:10.816296 29083 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 23:02:10.823720 29083 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 23:02:10.823864 29083 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 23:02:10.823947 29083 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 23:02:10.824047 29083 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 23:02:10.824124 29083 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 23:02:10.834214 29083 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786143361779964-25976-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786143361779964-25976-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43997
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786143361779964-25976-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43997
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786143361779964-25976-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 23:02:10.835561 29083 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 23:02:10.836910 29083 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 23:02:10.849095 29088 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 23:02:10.849146 29089 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 23:02:10.849260 29093 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 23:02:10.849924 29083 server_base.cc:1034] running on GCE node
I20260807 23:02:10.850776 29083 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 23:02:10.852212 29083 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 23:02:10.853463 29083 hybrid_clock.cc:648] HybridClock initialized: now 1786143730853407 us; error 48 us; skew 500 ppm
I20260807 23:02:10.853859 29083 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 23:02:10 dist-test-slave-z8x9 krb5kdc[29058](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143730, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 23:02:10.857285 29083 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 23:02:10.858896 29083 webserver.cc:460] Webserver started at http://127.0.0.1:36909/ using document root <none> and password file <none>
I20260807 23:02:10.859632 29083 fs_manager.cc:359] Metadata directory not provided
I20260807 23:02:10.859781 29083 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 23:02:10.860106 29083 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 23:02:10.863390 29083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786143361779964-25976-0/minicluster-data/master-0/data/instance:
uuid: "aed0149299e248ea9a5eb96c5918d50d"
format_stamp: "Formatted at 2026-08-07 23:02:10 on dist-test-slave-z8x9"
server_key: "48b2521847ae024cb266f3335f708f28"
server_key_iv: "8969a9d9f8fe551c381d1ec900f399d8"
server_key_version: "encryptionkey@0"
I20260807 23:02:10.864279 29083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786143361779964-25976-0/minicluster-data/master-0/wal/instance:
uuid: "aed0149299e248ea9a5eb96c5918d50d"
format_stamp: "Formatted at 2026-08-07 23:02:10 on dist-test-slave-z8x9"
server_key: "48b2521847ae024cb266f3335f708f28"
server_key_iv: "8969a9d9f8fe551c381d1ec900f399d8"
server_key_version: "encryptionkey@0"
I20260807 23:02:10.869822 29083 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260807 23:02:10.873478 29100 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 23:02:10.874789 29083 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 23:02:10.874962 29083 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786143361779964-25976-0/minicluster-data/master-0/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786143361779964-25976-0/minicluster-data/master-0/wal
uuid: "aed0149299e248ea9a5eb96c5918d50d"
format_stamp: "Formatted at 2026-08-07 23:02:10 on dist-test-slave-z8x9"
server_key: "48b2521847ae024cb266f3335f708f28"
server_key_iv: "8969a9d9f8fe551c381d1ec900f399d8"
server_key_version: "encryptionkey@0"
I20260807 23:02:10.875177 29083 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786143361779964-25976-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786143361779964-25976-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786143361779964-25976-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 23:02:10.892521 29083 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 23:02:10.897996 29083 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 23:02:10.898306 29083 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 23:02:10.915727 29083 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43997
I20260807 23:02:10.915766 29175 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43997 every 8 connection(s)
I20260807 23:02:10.917374 29083 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
I20260807 23:02:10.918255 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 29083
I20260807 23:02:10.918519 25976 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786143361779964-25976-0/minicluster-data/master-0/wal/instance
I20260807 23:02:10.918952 25976 external_mini_cluster.cc:1428] Setting key 629878326d842866984cd919755aa502
I20260807 23:02:10.921509 29176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 23:02:10.934983 29179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:10.920863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.69:52003 (local address 127.0.0.1:43997)
0807 23:02:10.922443 (+ 1580us) server_negotiation.cc:206] Beginning negotiation
0807 23:02:10.922457 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0807 23:02:10.922507 (+ 50us) server_negotiation.cc:407] Connection header received
0807 23:02:10.924675 (+ 2168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:02:10.924701 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:02:10.925240 (+ 539us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:02:10.925673 (+ 433us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:02:10.926186 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:10.928189 (+ 2003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:10.929774 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:10.930789 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:10.930982 (+ 193us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:02:10.932159 (+ 1177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:02:10.932170 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:02:10.932189 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 23:02:10.932938 (+ 749us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:02:10.933039 (+ 101us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 23:02:10.933074 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:02:10.933299 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0807 23:02:10.933516 (+ 217us) server_negotiation.cc:299] Negotiation successful
0807 23:02:10.933694 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1352,"thread_start_us":1057,"threads_started":1}
I20260807 23:02:10.936244 29176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aed0149299e248ea9a5eb96c5918d50d: Bootstrap starting.
I20260807 23:02:10.938714 25976 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 23:02:10.939560 29176 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aed0149299e248ea9a5eb96c5918d50d: Neither blocks nor log segments found. Creating new log.
I20260807 23:02:10.940544 29176 log.cc:824] T 00000000000000000000000000000000 P aed0149299e248ea9a5eb96c5918d50d: Log is configured to *not* fsync() on all Append() calls
I20260807 23:02:10.943440 29176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aed0149299e248ea9a5eb96c5918d50d: No bootstrap required, opened a new log
I20260807 23:02:10.952260 29176 raft_consensus.cc:348] T 00000000000000000000000000000000 P aed0149299e248ea9a5eb96c5918d50d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aed0149299e248ea9a5eb96c5918d50d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43997 } }
I20260807 23:02:10.952732 29176 raft_consensus.cc:374] T 00000000000000000000000000000000 P aed0149299e248ea9a5eb96c5918d50d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 23:02:10.952870 29176 raft_consensus.cc:729] T 00000000000000000000000000000000 P aed0149299e248ea9a5eb96c5918d50d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aed0149299e248ea9a5eb96c5918d50d, State: Initialized, Role: FOLLOWER
I20260807 23:02:10.953667 29176 consensus_queue.cc:260] T 00000000000000000000000000000000 P aed0149299e248ea9a5eb96c5918d50d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aed0149299e248ea9a5eb96c5918d50d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43997 } }
I20260807 23:02:10.953928 29176 raft_consensus.cc:388] T 00000000000000000000000000000000 P aed0149299e248ea9a5eb96c5918d50d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 23:02:10.954099 29176 raft_consensus.cc:482] T 00000000000000000000000000000000 P aed0149299e248ea9a5eb96c5918d50d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 23:02:10.954276 29176 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aed0149299e248ea9a5eb96c5918d50d [term 0 FOLLOWER]: Advancing to term 1
I20260807 23:02:10.955386 29179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:10.947943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.69:36653 (local address 127.0.0.1:43997)
0807 23:02:10.948121 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0807 23:02:10.948128 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 23:02:10.948656 (+ 528us) server_negotiation.cc:407] Connection header received
0807 23:02:10.948832 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:02:10.948836 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:02:10.948901 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:02:10.948978 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:02:10.949618 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:10.950442 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:10.952253 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:10.952938 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:10.953070 (+ 132us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:02:10.954598 (+ 1528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:02:10.954603 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:02:10.954606 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 23:02:10.954625 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:02:10.954662 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 23:02:10.954678 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:02:10.954800 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0807 23:02:10.955006 (+ 206us) server_negotiation.cc:299] Negotiation successful
0807 23:02:10.955067 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260807 23:02:10.957051 29176 raft_consensus.cc:504] T 00000000000000000000000000000000 P aed0149299e248ea9a5eb96c5918d50d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aed0149299e248ea9a5eb96c5918d50d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43997 } }
I20260807 23:02:10.957639 29176 leader_election.cc:302] T 00000000000000000000000000000000 P aed0149299e248ea9a5eb96c5918d50d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aed0149299e248ea9a5eb96c5918d50d; no voters:
I20260807 23:02:10.958246 29176 leader_election.cc:288] T 00000000000000000000000000000000 P aed0149299e248ea9a5eb96c5918d50d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 23:02:10.958384 29193 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aed0149299e248ea9a5eb96c5918d50d [term 1 FOLLOWER]: Leader election won for term 1
I20260807 23:02:10.959151 29193 raft_consensus.cc:686] T 00000000000000000000000000000000 P aed0149299e248ea9a5eb96c5918d50d [term 1 LEADER]: Becoming Leader. State: Replica: aed0149299e248ea9a5eb96c5918d50d, State: Running, Role: LEADER
I20260807 23:02:10.959812 29193 consensus_queue.cc:237] T 00000000000000000000000000000000 P aed0149299e248ea9a5eb96c5918d50d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aed0149299e248ea9a5eb96c5918d50d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43997 } }
I20260807 23:02:10.960484 29176 sys_catalog.cc:564] T 00000000000000000000000000000000 P aed0149299e248ea9a5eb96c5918d50d [sys.catalog]: configured and running, proceeding with master startup.
I20260807 23:02:10.962700 29195 sys_catalog.cc:455] T 00000000000000000000000000000000 P aed0149299e248ea9a5eb96c5918d50d [sys.catalog]: SysCatalogTable state changed. Reason: New leader aed0149299e248ea9a5eb96c5918d50d. Latest consensus state: current_term: 1 leader_uuid: "aed0149299e248ea9a5eb96c5918d50d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aed0149299e248ea9a5eb96c5918d50d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43997 } } }
I20260807 23:02:10.963097 29195 sys_catalog.cc:458] T 00000000000000000000000000000000 P aed0149299e248ea9a5eb96c5918d50d [sys.catalog]: This master's current role is: LEADER
I20260807 23:02:10.963591 29194 sys_catalog.cc:455] T 00000000000000000000000000000000 P aed0149299e248ea9a5eb96c5918d50d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aed0149299e248ea9a5eb96c5918d50d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aed0149299e248ea9a5eb96c5918d50d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43997 } } }
I20260807 23:02:10.964012 29194 sys_catalog.cc:458] T 00000000000000000000000000000000 P aed0149299e248ea9a5eb96c5918d50d [sys.catalog]: This master's current role is: LEADER
I20260807 23:02:10.964974 29200 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 23:02:10.970875 29200 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 23:02:10.980672 29200 catalog_manager.cc:1347] Generated new cluster ID: 1df361f4f5ce4ecc8126974446e13f0f
I20260807 23:02:10.980836 29200 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 23:02:11.000396 29200 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 23:02:11.001835 29200 catalog_manager.cc:1504] Loading token signing keys...
I20260807 23:02:11.010233 29200 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aed0149299e248ea9a5eb96c5918d50d: Generated new TSK 0
I20260807 23:02:11.011193 29200 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 23:02:11.018077 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 29083
W20260807 23:02:11.019274 29169 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.69:52003
W20260807 23:02:11.020079 25976 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 29083: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 23:02:11.020236 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 29083
W20260807 23:02:11.044737 29055 connection.cc:535] client connection to 127.0.0.1:43997 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43997): Connection reset by peer (error 104)
2026-08-07T23:02:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (2282 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786143361779964-25976-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 23:02:11 dist-test-slave-z8x9 krb5kdc[29224](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 23:02:11 dist-test-slave-z8x9 krb5kdc[29224](info): set up 2 sockets
Aug 07 23:02:11 dist-test-slave-z8x9 krb5kdc[29224](info): commencing operation
krb5kdc: starting...
W20260807 23:02:12.408093 25976 mini_kdc.cc:121] Time spent starting KDC: real 1.344s user 0.004s sys 0.045s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786143361779964-25976-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786143361779964-25976-0/krb5kdc/test-user.keytab.
Aug 07 23:02:12 dist-test-slave-z8x9 krb5kdc[29224](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143732, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T23:02:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T23:02:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786143361779964-25976-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786143361779964-25976-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 23:02:12.740728 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786143361779964-25976-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786143361779964-25976-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786143361779964-25976-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45381
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786143361779964-25976-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45381
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786143361779964-25976-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786143361779964-25976-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786143361779964-25976-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 23:02:13.095959 29244 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 23:02:13.096302 29244 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 23:02:13.096474 29244 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 23:02:13.103952 29244 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 23:02:13.104092 29244 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 23:02:13.104199 29244 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 23:02:13.104307 29244 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 23:02:13.104393 29244 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 23:02:13.114642 29244 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786143361779964-25976-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786143361779964-25976-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45381
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786143361779964-25976-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45381
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786143361779964-25976-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 23:02:13.116065 29244 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 23:02:13.117554 29244 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 23:02:13.130175 29251 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 23:02:13.130352 29253 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 23:02:13.130375 29250 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 23:02:13.131063 29244 server_base.cc:1034] running on GCE node
I20260807 23:02:13.131946 29244 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 23:02:13.133826 29244 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 23:02:13.135079 29244 hybrid_clock.cc:648] HybridClock initialized: now 1786143733135030 us; error 51 us; skew 500 ppm
I20260807 23:02:13.135479 29244 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 23:02:13 dist-test-slave-z8x9 krb5kdc[29224](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143733, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 23:02:13.144042 29244 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 23:02:13.146088 29244 webserver.cc:460] Webserver started at http://127.0.0.1:46815/ using document root <none> and password file <none>
I20260807 23:02:13.146898 29244 fs_manager.cc:359] Metadata directory not provided
I20260807 23:02:13.147069 29244 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 23:02:13.147424 29244 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 23:02:13.150791 29244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786143361779964-25976-0/minicluster-data/master-0/data/instance:
uuid: "f48f028c1d784a4483716f260bb1c517"
format_stamp: "Formatted at 2026-08-07 23:02:13 on dist-test-slave-z8x9"
server_key: "37d4d820916da980aede623cd009834d"
server_key_iv: "639e4d89204dd53d9a39f6b4b630d9d4"
server_key_version: "encryptionkey@0"
I20260807 23:02:13.151670 29244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786143361779964-25976-0/minicluster-data/master-0/wal/instance:
uuid: "f48f028c1d784a4483716f260bb1c517"
format_stamp: "Formatted at 2026-08-07 23:02:13 on dist-test-slave-z8x9"
server_key: "37d4d820916da980aede623cd009834d"
server_key_iv: "639e4d89204dd53d9a39f6b4b630d9d4"
server_key_version: "encryptionkey@0"
I20260807 23:02:13.157920 29244 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260807 23:02:13.162312 29264 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 23:02:13.164158 29244 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260807 23:02:13.164501 29244 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786143361779964-25976-0/minicluster-data/master-0/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786143361779964-25976-0/minicluster-data/master-0/wal
uuid: "f48f028c1d784a4483716f260bb1c517"
format_stamp: "Formatted at 2026-08-07 23:02:13 on dist-test-slave-z8x9"
server_key: "37d4d820916da980aede623cd009834d"
server_key_iv: "639e4d89204dd53d9a39f6b4b630d9d4"
server_key_version: "encryptionkey@0"
I20260807 23:02:13.164767 29244 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786143361779964-25976-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786143361779964-25976-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786143361779964-25976-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 23:02:13.171924 29244 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 23:02:13.177369 29244 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 23:02:13.177713 29244 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 23:02:13.201638 29244 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45381
I20260807 23:02:13.201700 29329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45381 every 8 connection(s)
I20260807 23:02:13.203222 29244 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
I20260807 23:02:13.207379 29330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 23:02:13.213637 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 29244
I20260807 23:02:13.213874 25976 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786143361779964-25976-0/minicluster-data/master-0/wal/instance
I20260807 23:02:13.214342 25976 external_mini_cluster.cc:1428] Setting key 1dfef20abb4783aa84f44816fa23a967
I20260807 23:02:13.219338 29330 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f48f028c1d784a4483716f260bb1c517: Bootstrap starting.
I20260807 23:02:13.222831 29334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:13.216281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.69:41751 (local address 127.0.0.1:45381)
0807 23:02:13.217162 (+ 881us) server_negotiation.cc:206] Beginning negotiation
0807 23:02:13.217241 (+ 79us) server_negotiation.cc:399] Waiting for connection header
0807 23:02:13.217292 (+ 51us) server_negotiation.cc:407] Connection header received
0807 23:02:13.219023 (+ 1731us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:02:13.219046 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:02:13.219405 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:02:13.219866 (+ 461us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:02:13.220989 (+ 1123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:02:13.220995 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:02:13.221013 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 23:02:13.221029 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:02:13.221100 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 23:02:13.221120 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:02:13.221444 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0807 23:02:13.221598 (+ 154us) server_negotiation.cc:299] Negotiation successful
0807 23:02:13.221794 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":618,"thread_start_us":368,"threads_started":1}
I20260807 23:02:13.223096 29330 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f48f028c1d784a4483716f260bb1c517: Neither blocks nor log segments found. Creating new log.
I20260807 23:02:13.224375 29330 log.cc:824] T 00000000000000000000000000000000 P f48f028c1d784a4483716f260bb1c517: Log is configured to *not* fsync() on all Append() calls
I20260807 23:02:13.227365 25976 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 23:02:13.227960 29330 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f48f028c1d784a4483716f260bb1c517: No bootstrap required, opened a new log
I20260807 23:02:13.235810 29330 raft_consensus.cc:348] T 00000000000000000000000000000000 P f48f028c1d784a4483716f260bb1c517 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f48f028c1d784a4483716f260bb1c517" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45381 } }
I20260807 23:02:13.236464 29330 raft_consensus.cc:374] T 00000000000000000000000000000000 P f48f028c1d784a4483716f260bb1c517 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 23:02:13.236601 29330 raft_consensus.cc:729] T 00000000000000000000000000000000 P f48f028c1d784a4483716f260bb1c517 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f48f028c1d784a4483716f260bb1c517, State: Initialized, Role: FOLLOWER
I20260807 23:02:13.237512 29330 consensus_queue.cc:260] T 00000000000000000000000000000000 P f48f028c1d784a4483716f260bb1c517 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f48f028c1d784a4483716f260bb1c517" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45381 } }
I20260807 23:02:13.237907 29330 raft_consensus.cc:388] T 00000000000000000000000000000000 P f48f028c1d784a4483716f260bb1c517 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 23:02:13.238095 29330 raft_consensus.cc:482] T 00000000000000000000000000000000 P f48f028c1d784a4483716f260bb1c517 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 23:02:13.238344 29330 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f48f028c1d784a4483716f260bb1c517 [term 0 FOLLOWER]: Advancing to term 1
I20260807 23:02:13.239431 29334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:13.236131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.69:56495 (local address 127.0.0.1:45381)
0807 23:02:13.236375 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0807 23:02:13.236383 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 23:02:13.236870 (+ 487us) server_negotiation.cc:407] Connection header received
0807 23:02:13.237109 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:02:13.237115 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:02:13.237254 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:02:13.237365 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:02:13.238650 (+ 1285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:02:13.238654 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:02:13.238658 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 23:02:13.238679 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:02:13.238727 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 23:02:13.238743 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:02:13.238859 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0807 23:02:13.238998 (+ 139us) server_negotiation.cc:299] Negotiation successful
0807 23:02:13.239056 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260807 23:02:13.241465 29330 raft_consensus.cc:504] T 00000000000000000000000000000000 P f48f028c1d784a4483716f260bb1c517 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f48f028c1d784a4483716f260bb1c517" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45381 } }
I20260807 23:02:13.242372 29330 leader_election.cc:302] T 00000000000000000000000000000000 P f48f028c1d784a4483716f260bb1c517 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f48f028c1d784a4483716f260bb1c517; no voters:
I20260807 23:02:13.243019 29330 leader_election.cc:288] T 00000000000000000000000000000000 P f48f028c1d784a4483716f260bb1c517 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 23:02:13.243199 29344 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f48f028c1d784a4483716f260bb1c517 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 23:02:13.243988 29344 raft_consensus.cc:686] T 00000000000000000000000000000000 P f48f028c1d784a4483716f260bb1c517 [term 1 LEADER]: Becoming Leader. State: Replica: f48f028c1d784a4483716f260bb1c517, State: Running, Role: LEADER
I20260807 23:02:13.244721 29344 consensus_queue.cc:237] T 00000000000000000000000000000000 P f48f028c1d784a4483716f260bb1c517 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f48f028c1d784a4483716f260bb1c517" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45381 } }
I20260807 23:02:13.245551 29330 sys_catalog.cc:564] T 00000000000000000000000000000000 P f48f028c1d784a4483716f260bb1c517 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 23:02:13.247841 29349 sys_catalog.cc:455] T 00000000000000000000000000000000 P f48f028c1d784a4483716f260bb1c517 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f48f028c1d784a4483716f260bb1c517. Latest consensus state: current_term: 1 leader_uuid: "f48f028c1d784a4483716f260bb1c517" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f48f028c1d784a4483716f260bb1c517" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45381 } } }
I20260807 23:02:13.248548 29349 sys_catalog.cc:458] T 00000000000000000000000000000000 P f48f028c1d784a4483716f260bb1c517 [sys.catalog]: This master's current role is: LEADER
I20260807 23:02:13.248131 29346 sys_catalog.cc:455] T 00000000000000000000000000000000 P f48f028c1d784a4483716f260bb1c517 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f48f028c1d784a4483716f260bb1c517" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f48f028c1d784a4483716f260bb1c517" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45381 } } }
I20260807 23:02:13.249118 29346 sys_catalog.cc:458] T 00000000000000000000000000000000 P f48f028c1d784a4483716f260bb1c517 [sys.catalog]: This master's current role is: LEADER
W20260807 23:02:13.259164 29373 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f48f028c1d784a4483716f260bb1c517: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 23:02:13.259336 29373 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 23:02:13.259518 29359 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 23:02:13.261385 29359 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 23:02:13.271708 29359 catalog_manager.cc:1347] Generated new cluster ID: 361aadfa6dc54612bfb74e1d3d1ebde1
I20260807 23:02:13.271857 29359 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 23:02:13.289579 29359 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 23:02:13.290127 29359 catalog_manager.cc:1504] Loading token signing keys...
I20260807 23:02:13.299909 29359 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f48f028c1d784a4483716f260bb1c517: Generated new TSK 0
I20260807 23:02:13.301132 29359 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 23:02:13.306074 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 29244
W20260807 23:02:13.307337 29325 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.69:41751
W20260807 23:02:13.308080 25976 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 29244: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 23:02:13.308249 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 29244
2026-08-07T23:02:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (2287 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786143361779964-25976-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 23:02:13 dist-test-slave-z8x9 krb5kdc[29383](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 23:02:13 dist-test-slave-z8x9 krb5kdc[29383](info): set up 2 sockets
Aug 07 23:02:13 dist-test-slave-z8x9 krb5kdc[29383](info): commencing operation
krb5kdc: starting...
W20260807 23:02:14.689441 25976 mini_kdc.cc:121] Time spent starting KDC: real 1.337s user 0.002s sys 0.044s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786143361779964-25976-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786143361779964-25976-0/krb5kdc/test-user.keytab.
Aug 07 23:02:14 dist-test-slave-z8x9 krb5kdc[29383](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143734, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T23:02:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T23:02:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.94.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.62.keytab.
Entry for principal kudu/127.25.94.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.62.keytab.
I20260807 23:02:15.020053 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786143361779964-25976-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786143361779964-25976-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786143361779964-25976-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.62:34973
--webserver_interface=127.25.94.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.62.keytab
--principal=kudu/127.25.94.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.20:43675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.94.62:34973
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786143361779964-25976-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786143361779964-25976-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786143361779964-25976-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 23:02:15.376376 29407 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 23:02:15.376662 29407 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 23:02:15.376817 29407 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 23:02:15.384301 29407 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 23:02:15.384415 29407 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 23:02:15.384524 29407 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 23:02:15.384621 29407 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 23:02:15.384703 29407 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 23:02:15.394804 29407 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:43675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786143361779964-25976-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786143361779964-25976-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.94.62:34973
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.62.keytab
--principal=kudu/127.25.94.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.62:34973
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.94.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786143361779964-25976-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 23:02:15.396158 29407 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 23:02:15.397547 29407 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 23:02:15.409832 29413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 23:02:15.409791 29415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 23:02:15.409802 29418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 23:02:15.410840 29407 server_base.cc:1034] running on GCE node
I20260807 23:02:15.411722 29407 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 23:02:15.413275 29407 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 23:02:15.414521 29407 hybrid_clock.cc:648] HybridClock initialized: now 1786143735414476 us; error 47 us; skew 500 ppm
I20260807 23:02:15.414901 29407 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 23:02:15 dist-test-slave-z8x9 krb5kdc[29383](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143735, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 23:02:15.418289 29407 init.cc:377] Logged in from keytab as kudu/127.25.94.62@KRBTEST.COM (short username kudu)
I20260807 23:02:15.419940 29407 webserver.cc:460] Webserver started at http://127.25.94.62:37713/ using document root <none> and password file <none>
I20260807 23:02:15.420679 29407 fs_manager.cc:359] Metadata directory not provided
I20260807 23:02:15.420842 29407 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 23:02:15.421229 29407 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 23:02:15.424396 29407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786143361779964-25976-0/minicluster-data/master-0/data/instance:
uuid: "fb0c68ccab1e40818f8f2f7b2f6ef955"
format_stamp: "Formatted at 2026-08-07 23:02:15 on dist-test-slave-z8x9"
server_key: "17034289f04240081ac83b5f4d1c8913"
server_key_iv: "6e7ac44193e8c0e2eb446f0462c9b0a6"
server_key_version: "encryptionkey@0"
I20260807 23:02:15.425290 29407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786143361779964-25976-0/minicluster-data/master-0/wal/instance:
uuid: "fb0c68ccab1e40818f8f2f7b2f6ef955"
format_stamp: "Formatted at 2026-08-07 23:02:15 on dist-test-slave-z8x9"
server_key: "17034289f04240081ac83b5f4d1c8913"
server_key_iv: "6e7ac44193e8c0e2eb446f0462c9b0a6"
server_key_version: "encryptionkey@0"
I20260807 23:02:15.430840 29407 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260807 23:02:15.434542 29431 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 23:02:15.435935 29407 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 23:02:15.436129 29407 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786143361779964-25976-0/minicluster-data/master-0/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786143361779964-25976-0/minicluster-data/master-0/wal
uuid: "fb0c68ccab1e40818f8f2f7b2f6ef955"
format_stamp: "Formatted at 2026-08-07 23:02:15 on dist-test-slave-z8x9"
server_key: "17034289f04240081ac83b5f4d1c8913"
server_key_iv: "6e7ac44193e8c0e2eb446f0462c9b0a6"
server_key_version: "encryptionkey@0"
I20260807 23:02:15.436347 29407 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786143361779964-25976-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786143361779964-25976-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786143361779964-25976-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 23:02:15.451103 29407 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 23:02:15.455996 29407 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 23:02:15.456329 29407 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 23:02:15.476261 29407 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.62:34973
I20260807 23:02:15.476316 29504 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.62:34973 every 8 connection(s)
I20260807 23:02:15.477849 29407 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
I20260807 23:02:15.481685 29505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 23:02:15.483768 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 29407
I20260807 23:02:15.483964 25976 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786143361779964-25976-0/minicluster-data/master-0/wal/instance
I20260807 23:02:15.484555 25976 external_mini_cluster.cc:1428] Setting key 3d2968a3da686a2230e211756736a339
I20260807 23:02:15.495515 29505 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb0c68ccab1e40818f8f2f7b2f6ef955: Bootstrap starting.
I20260807 23:02:15.499183 29505 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fb0c68ccab1e40818f8f2f7b2f6ef955: Neither blocks nor log segments found. Creating new log.
I20260807 23:02:15.500195 29505 log.cc:824] T 00000000000000000000000000000000 P fb0c68ccab1e40818f8f2f7b2f6ef955: Log is configured to *not* fsync() on all Append() calls
Aug 07 23:02:15 dist-test-slave-z8x9 krb5kdc[29383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143734, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.62@KRBTEST.COM
I20260807 23:02:15.503340 29505 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb0c68ccab1e40818f8f2f7b2f6ef955: No bootstrap required, opened a new log
I20260807 23:02:15.512074 29509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:15.486781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43945 (local address 127.25.94.62:34973)
0807 23:02:15.487843 (+ 1062us) server_negotiation.cc:206] Beginning negotiation
0807 23:02:15.487859 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0807 23:02:15.488223 (+ 364us) server_negotiation.cc:407] Connection header received
0807 23:02:15.490824 (+ 2601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:02:15.490856 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:02:15.491451 (+ 595us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:02:15.491946 (+ 495us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:02:15.493693 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:15.495736 (+ 2043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:15.497038 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:15.498126 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:15.498333 (+ 207us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:02:15.502976 (+ 4643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:02:15.502988 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:02:15.503008 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 23:02:15.503049 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:02:15.507246 (+ 4197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:02:15.508063 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:02:15.508071 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:02:15.508080 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:02:15.508312 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:02:15.508774 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:02:15.508779 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:02:15.508783 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:02:15.509545 (+ 762us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:02:15.509886 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0807 23:02:15.510329 (+ 443us) server_negotiation.cc:299] Negotiation successful
0807 23:02:15.510699 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":785,"thread_start_us":511,"threads_started":1}
I20260807 23:02:15.514424 29505 raft_consensus.cc:348] T 00000000000000000000000000000000 P fb0c68ccab1e40818f8f2f7b2f6ef955 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb0c68ccab1e40818f8f2f7b2f6ef955" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 34973 } }
I20260807 23:02:15.514943 29505 raft_consensus.cc:374] T 00000000000000000000000000000000 P fb0c68ccab1e40818f8f2f7b2f6ef955 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 23:02:15.515066 29505 raft_consensus.cc:729] T 00000000000000000000000000000000 P fb0c68ccab1e40818f8f2f7b2f6ef955 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb0c68ccab1e40818f8f2f7b2f6ef955, State: Initialized, Role: FOLLOWER
I20260807 23:02:15.515738 29505 consensus_queue.cc:260] T 00000000000000000000000000000000 P fb0c68ccab1e40818f8f2f7b2f6ef955 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb0c68ccab1e40818f8f2f7b2f6ef955" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 34973 } }
I20260807 23:02:15.516042 29505 raft_consensus.cc:388] T 00000000000000000000000000000000 P fb0c68ccab1e40818f8f2f7b2f6ef955 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 23:02:15.516214 29505 raft_consensus.cc:482] T 00000000000000000000000000000000 P fb0c68ccab1e40818f8f2f7b2f6ef955 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 23:02:15.516237 25976 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 23:02:15.516391 29505 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fb0c68ccab1e40818f8f2f7b2f6ef955 [term 0 FOLLOWER]: Advancing to term 1
I20260807 23:02:15.518662 29505 raft_consensus.cc:504] T 00000000000000000000000000000000 P fb0c68ccab1e40818f8f2f7b2f6ef955 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb0c68ccab1e40818f8f2f7b2f6ef955" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 34973 } }
I20260807 23:02:15.519181 29505 leader_election.cc:302] T 00000000000000000000000000000000 P fb0c68ccab1e40818f8f2f7b2f6ef955 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb0c68ccab1e40818f8f2f7b2f6ef955; no voters:
I20260807 23:02:15.519868 29505 leader_election.cc:288] T 00000000000000000000000000000000 P fb0c68ccab1e40818f8f2f7b2f6ef955 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 23:02:15.519970 29516 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fb0c68ccab1e40818f8f2f7b2f6ef955 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 23:02:15.522599 29505 sys_catalog.cc:564] T 00000000000000000000000000000000 P fb0c68ccab1e40818f8f2f7b2f6ef955 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 23:02:15.525928 29516 raft_consensus.cc:686] T 00000000000000000000000000000000 P fb0c68ccab1e40818f8f2f7b2f6ef955 [term 1 LEADER]: Becoming Leader. State: Replica: fb0c68ccab1e40818f8f2f7b2f6ef955, State: Running, Role: LEADER
I20260807 23:02:15.526933 29516 consensus_queue.cc:237] T 00000000000000000000000000000000 P fb0c68ccab1e40818f8f2f7b2f6ef955 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb0c68ccab1e40818f8f2f7b2f6ef955" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 34973 } }
I20260807 23:02:15.533263 29518 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb0c68ccab1e40818f8f2f7b2f6ef955 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb0c68ccab1e40818f8f2f7b2f6ef955. Latest consensus state: current_term: 1 leader_uuid: "fb0c68ccab1e40818f8f2f7b2f6ef955" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb0c68ccab1e40818f8f2f7b2f6ef955" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 34973 } } }
I20260807 23:02:15.533680 29518 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb0c68ccab1e40818f8f2f7b2f6ef955 [sys.catalog]: This master's current role is: LEADER
I20260807 23:02:15.534305 29517 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb0c68ccab1e40818f8f2f7b2f6ef955 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fb0c68ccab1e40818f8f2f7b2f6ef955" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb0c68ccab1e40818f8f2f7b2f6ef955" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 34973 } } }
I20260807 23:02:15.534646 29517 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb0c68ccab1e40818f8f2f7b2f6ef955 [sys.catalog]: This master's current role is: LEADER
W20260807 23:02:15.542635 29541 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fb0c68ccab1e40818f8f2f7b2f6ef955: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 23:02:15.542810 29541 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 23:02:15.542970 29538 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 23:02:15.545320 29538 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 23:02:15.545667 29509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:15.521359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38345 (local address 127.25.94.62:34973)
0807 23:02:15.521591 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0807 23:02:15.521600 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 23:02:15.525843 (+ 4243us) server_negotiation.cc:407] Connection header received
0807 23:02:15.526161 (+ 318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:02:15.526169 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:02:15.526291 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:02:15.526387 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:02:15.528636 (+ 2249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:15.530466 (+ 1830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:15.532928 (+ 2462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:15.533970 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:15.534116 (+ 146us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:02:15.537834 (+ 3718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:02:15.537841 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:02:15.537846 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 23:02:15.537887 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:02:15.541853 (+ 3966us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:02:15.542905 (+ 1052us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:02:15.542912 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:02:15.542917 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:02:15.543097 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:02:15.543711 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:02:15.543716 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:02:15.543719 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:02:15.544188 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:02:15.544492 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0807 23:02:15.544991 (+ 499us) server_negotiation.cc:299] Negotiation successful
0807 23:02:15.545235 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260807 23:02:15.554513 29538 catalog_manager.cc:1347] Generated new cluster ID: e8abd927ceb44e73861e86557f0c3caf
I20260807 23:02:15.554702 29538 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 23:02:15.568229 29538 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 23:02:15.569890 29538 catalog_manager.cc:1504] Loading token signing keys...
I20260807 23:02:15.586545 29538 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fb0c68ccab1e40818f8f2f7b2f6ef955: Generated new TSK 0
I20260807 23:02:15.587566 29538 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 23:02:15.592912 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 29407
I20260807 23:02:15.594247 29499 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 23:02:15.686156 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 29407
W20260807 23:02:15.711390 29380 connection.cc:535] client connection to 127.25.94.62:34973 recv error: Network error: failed to read from TLS socket (remote: 127.25.94.62:34973): Connection reset by peer (error 104)
2026-08-07T23:02:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2378 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786143361779964-25976-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 23:02:15 dist-test-slave-z8x9 krb5kdc[29551](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 23:02:15 dist-test-slave-z8x9 krb5kdc[29551](info): set up 2 sockets
Aug 07 23:02:15 dist-test-slave-z8x9 krb5kdc[29551](info): commencing operation
krb5kdc: starting...
W20260807 23:02:17.063606 25976 mini_kdc.cc:121] Time spent starting KDC: real 1.333s user 0.003s sys 0.041s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786143361779964-25976-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786143361779964-25976-0/krb5kdc/test-user.keytab.
Aug 07 23:02:17 dist-test-slave-z8x9 krb5kdc[29551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143737, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T23:02:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T23:02:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.94.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.62.keytab.
Entry for principal kudu/127.25.94.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.62.keytab.
I20260807 23:02:17.410024 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786143361779964-25976-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786143361779964-25976-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786143361779964-25976-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.62:41855
--webserver_interface=127.25.94.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.62.keytab
--principal=kudu/127.25.94.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.20:34713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.94.62:41855
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786143361779964-25976-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786143361779964-25976-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786143361779964-25976-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 23:02:17.756209 29566 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 23:02:17.756518 29566 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 23:02:17.756719 29566 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 23:02:17.764135 29566 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 23:02:17.764258 29566 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 23:02:17.764376 29566 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 23:02:17.764472 29566 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 23:02:17.764537 29566 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 23:02:17.774600 29566 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:34713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786143361779964-25976-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786143361779964-25976-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.94.62:41855
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.62.keytab
--principal=kudu/127.25.94.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.62:41855
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.94.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786143361779964-25976-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 23:02:17.775965 29566 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 23:02:17.777333 29566 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 23:02:17.789673 29573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 23:02:17.789677 29574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 23:02:17.789892 29576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 23:02:17.790997 29566 server_base.cc:1034] running on GCE node
I20260807 23:02:17.791872 29566 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 23:02:17.793355 29566 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 23:02:17.794564 29566 hybrid_clock.cc:648] HybridClock initialized: now 1786143737794506 us; error 53 us; skew 500 ppm
I20260807 23:02:17.794931 29566 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 23:02:17 dist-test-slave-z8x9 krb5kdc[29551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143737, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 23:02:17.798200 29566 init.cc:377] Logged in from keytab as kudu/127.25.94.62@KRBTEST.COM (short username kudu)
I20260807 23:02:17.799650 29566 webserver.cc:460] Webserver started at http://127.25.94.62:44663/ using document root <none> and password file <none>
I20260807 23:02:17.800292 29566 fs_manager.cc:359] Metadata directory not provided
I20260807 23:02:17.800391 29566 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 23:02:17.800639 29566 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 23:02:17.803673 29566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786143361779964-25976-0/minicluster-data/master-0/data/instance:
uuid: "3fbba52e74f04c57bb8b3ff49aecf035"
format_stamp: "Formatted at 2026-08-07 23:02:17 on dist-test-slave-z8x9"
server_key: "274da1ffe18e6d0312820c5b57230572"
server_key_iv: "648daba2d22b089b819a6bb773171960"
server_key_version: "encryptionkey@0"
I20260807 23:02:17.804399 29566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786143361779964-25976-0/minicluster-data/master-0/wal/instance:
uuid: "3fbba52e74f04c57bb8b3ff49aecf035"
format_stamp: "Formatted at 2026-08-07 23:02:17 on dist-test-slave-z8x9"
server_key: "274da1ffe18e6d0312820c5b57230572"
server_key_iv: "648daba2d22b089b819a6bb773171960"
server_key_version: "encryptionkey@0"
I20260807 23:02:17.809644 29566 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260807 23:02:17.813110 29593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 23:02:17.814494 29566 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 23:02:17.814683 29566 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786143361779964-25976-0/minicluster-data/master-0/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786143361779964-25976-0/minicluster-data/master-0/wal
uuid: "3fbba52e74f04c57bb8b3ff49aecf035"
format_stamp: "Formatted at 2026-08-07 23:02:17 on dist-test-slave-z8x9"
server_key: "274da1ffe18e6d0312820c5b57230572"
server_key_iv: "648daba2d22b089b819a6bb773171960"
server_key_version: "encryptionkey@0"
I20260807 23:02:17.814888 29566 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786143361779964-25976-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786143361779964-25976-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786143361779964-25976-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 23:02:17.842373 29566 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 23:02:17.846613 29566 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 23:02:17.846936 29566 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 23:02:17.867632 29566 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.62:41855
I20260807 23:02:17.867664 29676 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.62:41855 every 8 connection(s)
I20260807 23:02:17.869225 29566 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
I20260807 23:02:17.873050 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 29566
I20260807 23:02:17.873315 25976 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786143361779964-25976-0/minicluster-data/master-0/wal/instance
I20260807 23:02:17.873440 29677 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 23:02:17.873876 25976 external_mini_cluster.cc:1428] Setting key 0d678bd5cba4472938a826717d092f58
I20260807 23:02:17.889554 29682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:17.875955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41729 (local address 127.25.94.62:41855)
0807 23:02:17.877033 (+ 1078us) server_negotiation.cc:206] Beginning negotiation
0807 23:02:17.877049 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0807 23:02:17.877106 (+ 57us) server_negotiation.cc:407] Connection header received
0807 23:02:17.878954 (+ 1848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:02:17.878982 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:02:17.879558 (+ 576us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:02:17.880036 (+ 478us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:02:17.880602 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:17.882683 (+ 2081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:17.884089 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:17.885142 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:17.885352 (+ 210us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:02:17.886675 (+ 1323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:02:17.886686 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:02:17.886711 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 23:02:17.887480 (+ 769us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:02:17.887579 (+ 99us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 23:02:17.887609 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:02:17.887755 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0807 23:02:17.888047 (+ 292us) server_negotiation.cc:299] Negotiation successful
0807 23:02:17.888229 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":821,"thread_start_us":583,"threads_started":1}
I20260807 23:02:17.889698 29677 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3fbba52e74f04c57bb8b3ff49aecf035: Bootstrap starting.
I20260807 23:02:17.892971 29677 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3fbba52e74f04c57bb8b3ff49aecf035: Neither blocks nor log segments found. Creating new log.
I20260807 23:02:17.893908 25976 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 23:02:17.894410 29677 log.cc:824] T 00000000000000000000000000000000 P 3fbba52e74f04c57bb8b3ff49aecf035: Log is configured to *not* fsync() on all Append() calls
I20260807 23:02:17.897550 29677 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3fbba52e74f04c57bb8b3ff49aecf035: No bootstrap required, opened a new log
I20260807 23:02:17.905987 29682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:17.898724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46459 (local address 127.25.94.62:41855)
0807 23:02:17.898950 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0807 23:02:17.898957 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 23:02:17.899468 (+ 511us) server_negotiation.cc:407] Connection header received
0807 23:02:17.899647 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:02:17.899651 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:02:17.899710 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:02:17.899796 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:02:17.900405 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:17.901215 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:17.903177 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:17.903681 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:17.903794 (+ 113us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:02:17.905086 (+ 1292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:02:17.905091 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:02:17.905094 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 23:02:17.905112 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:02:17.905216 (+ 104us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 23:02:17.905245 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:02:17.905335 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0807 23:02:17.905555 (+ 220us) server_negotiation.cc:299] Negotiation successful
0807 23:02:17.905599 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260807 23:02:17.907832 29677 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3fbba52e74f04c57bb8b3ff49aecf035 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fbba52e74f04c57bb8b3ff49aecf035" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 41855 } }
I20260807 23:02:17.908262 29677 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3fbba52e74f04c57bb8b3ff49aecf035 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 23:02:17.908377 29677 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3fbba52e74f04c57bb8b3ff49aecf035 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fbba52e74f04c57bb8b3ff49aecf035, State: Initialized, Role: FOLLOWER
I20260807 23:02:17.909097 29677 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3fbba52e74f04c57bb8b3ff49aecf035 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fbba52e74f04c57bb8b3ff49aecf035" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 41855 } }
I20260807 23:02:17.909472 29677 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3fbba52e74f04c57bb8b3ff49aecf035 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 23:02:17.909641 29677 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3fbba52e74f04c57bb8b3ff49aecf035 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 23:02:17.909860 29677 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3fbba52e74f04c57bb8b3ff49aecf035 [term 0 FOLLOWER]: Advancing to term 1
I20260807 23:02:17.912201 29677 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3fbba52e74f04c57bb8b3ff49aecf035 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fbba52e74f04c57bb8b3ff49aecf035" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 41855 } }
I20260807 23:02:17.912855 29677 leader_election.cc:302] T 00000000000000000000000000000000 P 3fbba52e74f04c57bb8b3ff49aecf035 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3fbba52e74f04c57bb8b3ff49aecf035; no voters:
I20260807 23:02:17.913656 29677 leader_election.cc:288] T 00000000000000000000000000000000 P 3fbba52e74f04c57bb8b3ff49aecf035 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 23:02:17.913760 29692 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3fbba52e74f04c57bb8b3ff49aecf035 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 23:02:17.914557 29692 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3fbba52e74f04c57bb8b3ff49aecf035 [term 1 LEADER]: Becoming Leader. State: Replica: 3fbba52e74f04c57bb8b3ff49aecf035, State: Running, Role: LEADER
I20260807 23:02:17.915162 29692 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3fbba52e74f04c57bb8b3ff49aecf035 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fbba52e74f04c57bb8b3ff49aecf035" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 41855 } }
I20260807 23:02:17.915962 29677 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3fbba52e74f04c57bb8b3ff49aecf035 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 23:02:17.918181 29694 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3fbba52e74f04c57bb8b3ff49aecf035 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3fbba52e74f04c57bb8b3ff49aecf035. Latest consensus state: current_term: 1 leader_uuid: "3fbba52e74f04c57bb8b3ff49aecf035" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fbba52e74f04c57bb8b3ff49aecf035" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 41855 } } }
I20260807 23:02:17.918676 29694 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3fbba52e74f04c57bb8b3ff49aecf035 [sys.catalog]: This master's current role is: LEADER
I20260807 23:02:17.919291 29693 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3fbba52e74f04c57bb8b3ff49aecf035 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3fbba52e74f04c57bb8b3ff49aecf035" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fbba52e74f04c57bb8b3ff49aecf035" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 41855 } } }
I20260807 23:02:17.919610 29693 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3fbba52e74f04c57bb8b3ff49aecf035 [sys.catalog]: This master's current role is: LEADER
I20260807 23:02:17.921433 29701 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 23:02:17.928397 29701 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 23:02:17.937904 29701 catalog_manager.cc:1347] Generated new cluster ID: ec7ca126dffd40d68d610d3ba63404db
I20260807 23:02:17.938038 29701 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 23:02:17.959074 29701 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 23:02:17.960443 29701 catalog_manager.cc:1504] Loading token signing keys...
I20260807 23:02:17.966913 29701 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3fbba52e74f04c57bb8b3ff49aecf035: Generated new TSK 0
I20260807 23:02:17.967818 29701 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 23:02:17.985350 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 29566
W20260807 23:02:17.987169 29669 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:41729
W20260807 23:02:17.987998 25976 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 29566: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 23:02:17.988162 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 29566
W20260807 23:02:18.012322 29548 connection.cc:535] client connection to 127.25.94.62:41855 recv error: Network error: failed to read from TLS socket (remote: 127.25.94.62:41855): Connection reset by peer (error 104)
2026-08-07T23:02:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (2301 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786143361779964-25976-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 23:02:18 dist-test-slave-z8x9 krb5kdc[29724](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 23:02:18 dist-test-slave-z8x9 krb5kdc[29724](info): set up 2 sockets
Aug 07 23:02:18 dist-test-slave-z8x9 krb5kdc[29724](info): commencing operation
krb5kdc: starting...
W20260807 23:02:19.395447 25976 mini_kdc.cc:121] Time spent starting KDC: real 1.363s user 0.000s sys 0.050s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786143361779964-25976-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786143361779964-25976-0/krb5kdc/test-user.keytab.
Aug 07 23:02:19 dist-test-slave-z8x9 krb5kdc[29724](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143739, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T23:02:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T23:02:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.94.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.62.keytab.
Entry for principal kudu/127.25.94.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.62.keytab.
I20260807 23:02:19.737914 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786143361779964-25976-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786143361779964-25976-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786143361779964-25976-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.62:39681
--webserver_interface=127.25.94.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.62.keytab
--principal=kudu/127.25.94.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.20:33017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.94.62:39681
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786143361779964-25976-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786143361779964-25976-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786143361779964-25976-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 23:02:20.088423 29746 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 23:02:20.088779 29746 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 23:02:20.088955 29746 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 23:02:20.097093 29746 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 23:02:20.097334 29746 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 23:02:20.097472 29746 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 23:02:20.097561 29746 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 23:02:20.097643 29746 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 23:02:20.108939 29746 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:33017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786143361779964-25976-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786143361779964-25976-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.94.62:39681
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.62.keytab
--principal=kudu/127.25.94.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.62:39681
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.94.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786143361779964-25976-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 23:02:20.110503 29746 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 23:02:20.112257 29746 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 23:02:20.126611 29754 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 23:02:20.126587 29761 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 23:02:20.126629 29756 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 23:02:20.127942 29746 server_base.cc:1034] running on GCE node
I20260807 23:02:20.128988 29746 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 23:02:20.130759 29746 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 23:02:20.132041 29746 hybrid_clock.cc:648] HybridClock initialized: now 1786143740131997 us; error 45 us; skew 500 ppm
I20260807 23:02:20.132457 29746 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 23:02:20 dist-test-slave-z8x9 krb5kdc[29724](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143740, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 23:02:20.136488 29746 init.cc:377] Logged in from keytab as kudu/127.25.94.62@KRBTEST.COM (short username kudu)
I20260807 23:02:20.138324 29746 webserver.cc:460] Webserver started at http://127.25.94.62:36735/ using document root <none> and password file <none>
I20260807 23:02:20.139117 29746 fs_manager.cc:359] Metadata directory not provided
I20260807 23:02:20.139288 29746 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 23:02:20.139637 29746 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 23:02:20.143203 29746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786143361779964-25976-0/minicluster-data/master-0/data/instance:
uuid: "f8b06899aa2c4e8da907d2af26cbe6d2"
format_stamp: "Formatted at 2026-08-07 23:02:20 on dist-test-slave-z8x9"
server_key: "03faf2494759da0e795e3e78375f8228"
server_key_iv: "0d932ee2d03843ee42e6a05e3e4f1a9c"
server_key_version: "encryptionkey@0"
I20260807 23:02:20.144073 29746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786143361779964-25976-0/minicluster-data/master-0/wal/instance:
uuid: "f8b06899aa2c4e8da907d2af26cbe6d2"
format_stamp: "Formatted at 2026-08-07 23:02:20 on dist-test-slave-z8x9"
server_key: "03faf2494759da0e795e3e78375f8228"
server_key_iv: "0d932ee2d03843ee42e6a05e3e4f1a9c"
server_key_version: "encryptionkey@0"
I20260807 23:02:20.150070 29746 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260807 23:02:20.154599 29771 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 23:02:20.156628 29746 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260807 23:02:20.156917 29746 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786143361779964-25976-0/minicluster-data/master-0/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786143361779964-25976-0/minicluster-data/master-0/wal
uuid: "f8b06899aa2c4e8da907d2af26cbe6d2"
format_stamp: "Formatted at 2026-08-07 23:02:20 on dist-test-slave-z8x9"
server_key: "03faf2494759da0e795e3e78375f8228"
server_key_iv: "0d932ee2d03843ee42e6a05e3e4f1a9c"
server_key_version: "encryptionkey@0"
I20260807 23:02:20.157259 29746 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786143361779964-25976-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786143361779964-25976-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786143361779964-25976-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 23:02:20.163954 29746 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 23:02:20.169718 29746 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 23:02:20.170084 29746 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 23:02:20.192476 29746 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.62:39681
I20260807 23:02:20.192476 29860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.62:39681 every 8 connection(s)
I20260807 23:02:20.194204 29746 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
I20260807 23:02:20.198396 29866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 23:02:20.201334 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 29746
I20260807 23:02:20.201531 25976 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786143361779964-25976-0/minicluster-data/master-0/wal/instance
I20260807 23:02:20.202092 25976 external_mini_cluster.cc:1428] Setting key 29d0d8636d73f024537414521d75a802
I20260807 23:02:20.211376 29871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:20.204169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49241 (local address 127.25.94.62:39681)
0807 23:02:20.205037 (+ 868us) server_negotiation.cc:206] Beginning negotiation
0807 23:02:20.205049 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 23:02:20.205092 (+ 43us) server_negotiation.cc:407] Connection header received
0807 23:02:20.206898 (+ 1806us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:02:20.206921 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:02:20.207280 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:02:20.207726 (+ 446us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:02:20.209548 (+ 1822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:02:20.209555 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:02:20.209569 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 23:02:20.209588 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:02:20.209661 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 23:02:20.209683 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:02:20.210030 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0807 23:02:20.210176 (+ 146us) server_negotiation.cc:299] Negotiation successful
0807 23:02:20.210360 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":642,"thread_start_us":373,"threads_started":1}
I20260807 23:02:20.212831 29866 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8b06899aa2c4e8da907d2af26cbe6d2: Bootstrap starting.
I20260807 23:02:20.215552 25976 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 23:02:20.216725 29866 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f8b06899aa2c4e8da907d2af26cbe6d2: Neither blocks nor log segments found. Creating new log.
I20260807 23:02:20.217964 29866 log.cc:824] T 00000000000000000000000000000000 P f8b06899aa2c4e8da907d2af26cbe6d2: Log is configured to *not* fsync() on all Append() calls
I20260807 23:02:20.221640 29866 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8b06899aa2c4e8da907d2af26cbe6d2: No bootstrap required, opened a new log
I20260807 23:02:20.223529 29871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:20.220555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60363 (local address 127.25.94.62:39681)
0807 23:02:20.220718 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0807 23:02:20.220722 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 23:02:20.221322 (+ 600us) server_negotiation.cc:407] Connection header received
0807 23:02:20.221546 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:02:20.221550 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:02:20.221601 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:02:20.221712 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:02:20.222855 (+ 1143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:02:20.222861 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:02:20.222864 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 23:02:20.222881 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:02:20.222928 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 23:02:20.222945 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:02:20.223080 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0807 23:02:20.223170 (+ 90us) server_negotiation.cc:299] Negotiation successful
0807 23:02:20.223234 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260807 23:02:20.231096 29866 raft_consensus.cc:348] T 00000000000000000000000000000000 P f8b06899aa2c4e8da907d2af26cbe6d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8b06899aa2c4e8da907d2af26cbe6d2" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 39681 } }
I20260807 23:02:20.231566 29866 raft_consensus.cc:374] T 00000000000000000000000000000000 P f8b06899aa2c4e8da907d2af26cbe6d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 23:02:20.231684 29866 raft_consensus.cc:729] T 00000000000000000000000000000000 P f8b06899aa2c4e8da907d2af26cbe6d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8b06899aa2c4e8da907d2af26cbe6d2, State: Initialized, Role: FOLLOWER
I20260807 23:02:20.232312 29866 consensus_queue.cc:260] T 00000000000000000000000000000000 P f8b06899aa2c4e8da907d2af26cbe6d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8b06899aa2c4e8da907d2af26cbe6d2" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 39681 } }
I20260807 23:02:20.232590 29866 raft_consensus.cc:388] T 00000000000000000000000000000000 P f8b06899aa2c4e8da907d2af26cbe6d2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 23:02:20.232757 29866 raft_consensus.cc:482] T 00000000000000000000000000000000 P f8b06899aa2c4e8da907d2af26cbe6d2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 23:02:20.232946 29866 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f8b06899aa2c4e8da907d2af26cbe6d2 [term 0 FOLLOWER]: Advancing to term 1
I20260807 23:02:20.235417 29866 raft_consensus.cc:504] T 00000000000000000000000000000000 P f8b06899aa2c4e8da907d2af26cbe6d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8b06899aa2c4e8da907d2af26cbe6d2" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 39681 } }
I20260807 23:02:20.235971 29866 leader_election.cc:302] T 00000000000000000000000000000000 P f8b06899aa2c4e8da907d2af26cbe6d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f8b06899aa2c4e8da907d2af26cbe6d2; no voters:
I20260807 23:02:20.236769 29866 leader_election.cc:288] T 00000000000000000000000000000000 P f8b06899aa2c4e8da907d2af26cbe6d2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 23:02:20.236974 29884 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f8b06899aa2c4e8da907d2af26cbe6d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 23:02:20.237993 29884 raft_consensus.cc:686] T 00000000000000000000000000000000 P f8b06899aa2c4e8da907d2af26cbe6d2 [term 1 LEADER]: Becoming Leader. State: Replica: f8b06899aa2c4e8da907d2af26cbe6d2, State: Running, Role: LEADER
I20260807 23:02:20.238536 29884 consensus_queue.cc:237] T 00000000000000000000000000000000 P f8b06899aa2c4e8da907d2af26cbe6d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8b06899aa2c4e8da907d2af26cbe6d2" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 39681 } }
I20260807 23:02:20.239130 29866 sys_catalog.cc:564] T 00000000000000000000000000000000 P f8b06899aa2c4e8da907d2af26cbe6d2 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 23:02:20.241717 29888 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8b06899aa2c4e8da907d2af26cbe6d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f8b06899aa2c4e8da907d2af26cbe6d2. Latest consensus state: current_term: 1 leader_uuid: "f8b06899aa2c4e8da907d2af26cbe6d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8b06899aa2c4e8da907d2af26cbe6d2" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 39681 } } }
I20260807 23:02:20.241668 29887 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8b06899aa2c4e8da907d2af26cbe6d2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f8b06899aa2c4e8da907d2af26cbe6d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8b06899aa2c4e8da907d2af26cbe6d2" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 39681 } } }
I20260807 23:02:20.242252 29887 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8b06899aa2c4e8da907d2af26cbe6d2 [sys.catalog]: This master's current role is: LEADER
I20260807 23:02:20.242944 29888 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8b06899aa2c4e8da907d2af26cbe6d2 [sys.catalog]: This master's current role is: LEADER
I20260807 23:02:20.244442 29899 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 23:02:20.250229 29899 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 23:02:20.260985 29899 catalog_manager.cc:1347] Generated new cluster ID: d99a963f003b44dfa0b4d16cbb5aeda1
I20260807 23:02:20.261148 29899 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 23:02:20.277688 29899 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 23:02:20.278199 29899 catalog_manager.cc:1504] Loading token signing keys...
I20260807 23:02:20.286430 29899 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f8b06899aa2c4e8da907d2af26cbe6d2: Generated new TSK 0
I20260807 23:02:20.287304 29899 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 23:02:20.295415 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 29746
W20260807 23:02:20.296993 29854 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:49241
W20260807 23:02:20.297863 25976 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 29746: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 23:02:20.298033 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 29746
2026-08-07T23:02:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (2319 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786143361779964-25976-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 23:02:20 dist-test-slave-z8x9 krb5kdc[29918](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 23:02:20 dist-test-slave-z8x9 krb5kdc[29918](info): set up 2 sockets
Aug 07 23:02:20 dist-test-slave-z8x9 krb5kdc[29918](info): commencing operation
krb5kdc: starting...
W20260807 23:02:21.682478 25976 mini_kdc.cc:121] Time spent starting KDC: real 1.331s user 0.000s sys 0.045s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786143361779964-25976-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786143361779964-25976-0/krb5kdc/test-user.keytab.
Aug 07 23:02:21 dist-test-slave-z8x9 krb5kdc[29918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143741, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T23:02:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T23:02:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.94.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.62.keytab.
Entry for principal kudu/127.25.94.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.62.keytab.
I20260807 23:02:22.020979 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786143361779964-25976-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786143361779964-25976-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786143361779964-25976-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.62:44841
--webserver_interface=127.25.94.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.62.keytab
--principal=kudu/127.25.94.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.20:43197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.94.62:44841
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786143361779964-25976-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786143361779964-25976-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786143361779964-25976-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 23:02:22.378090 29943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 23:02:22.378494 29943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 23:02:22.378666 29943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 23:02:22.386314 29943 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 23:02:22.386500 29943 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 23:02:22.386618 29943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 23:02:22.386742 29943 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 23:02:22.386832 29943 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 23:02:22.397534 29943 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:43197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786143361779964-25976-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786143361779964-25976-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.94.62:44841
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.62.keytab
--principal=kudu/127.25.94.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.62:44841
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.94.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786143361779964-25976-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 23:02:22.398988 29943 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 23:02:22.400322 29943 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 23:02:22.412487 29949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 23:02:22.412570 29950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 23:02:22.412529 29953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 23:02:22.413756 29943 server_base.cc:1034] running on GCE node
I20260807 23:02:22.414566 29943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 23:02:22.416024 29943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 23:02:22.417279 29943 hybrid_clock.cc:648] HybridClock initialized: now 1786143742417229 us; error 58 us; skew 500 ppm
I20260807 23:02:22.417665 29943 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 23:02:22 dist-test-slave-z8x9 krb5kdc[29918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143742, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 23:02:22.421232 29943 init.cc:377] Logged in from keytab as kudu/127.25.94.62@KRBTEST.COM (short username kudu)
I20260807 23:02:22.422935 29943 webserver.cc:460] Webserver started at http://127.25.94.62:34035/ using document root <none> and password file <none>
I20260807 23:02:22.423668 29943 fs_manager.cc:359] Metadata directory not provided
I20260807 23:02:22.423821 29943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 23:02:22.424136 29943 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 23:02:22.427206 29943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786143361779964-25976-0/minicluster-data/master-0/data/instance:
uuid: "76d4ae1889544a16a83819222754a5f9"
format_stamp: "Formatted at 2026-08-07 23:02:22 on dist-test-slave-z8x9"
server_key: "e9d9d9188e8a63e7c897f3d412df41ff"
server_key_iv: "5371c1311a5fc7674bef52bbaa2140be"
server_key_version: "encryptionkey@0"
I20260807 23:02:22.427996 29943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786143361779964-25976-0/minicluster-data/master-0/wal/instance:
uuid: "76d4ae1889544a16a83819222754a5f9"
format_stamp: "Formatted at 2026-08-07 23:02:22 on dist-test-slave-z8x9"
server_key: "e9d9d9188e8a63e7c897f3d412df41ff"
server_key_iv: "5371c1311a5fc7674bef52bbaa2140be"
server_key_version: "encryptionkey@0"
I20260807 23:02:22.433531 29943 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260807 23:02:22.436973 29961 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 23:02:22.438347 29943 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 23:02:22.438561 29943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786143361779964-25976-0/minicluster-data/master-0/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786143361779964-25976-0/minicluster-data/master-0/wal
uuid: "76d4ae1889544a16a83819222754a5f9"
format_stamp: "Formatted at 2026-08-07 23:02:22 on dist-test-slave-z8x9"
server_key: "e9d9d9188e8a63e7c897f3d412df41ff"
server_key_iv: "5371c1311a5fc7674bef52bbaa2140be"
server_key_version: "encryptionkey@0"
I20260807 23:02:22.438777 29943 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786143361779964-25976-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786143361779964-25976-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786143361779964-25976-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 23:02:22.487246 29943 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 23:02:22.491649 29943 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 23:02:22.491981 29943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 23:02:22.512672 29943 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.62:44841
I20260807 23:02:22.512704 30043 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.62:44841 every 8 connection(s)
I20260807 23:02:22.514134 29943 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
I20260807 23:02:22.515803 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 29943
I20260807 23:02:22.516013 25976 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786143361779964-25976-0/minicluster-data/master-0/wal/instance
I20260807 23:02:22.516603 25976 external_mini_cluster.cc:1428] Setting key c3f3f332a4a049cde2bdd9fe38f56bd5
I20260807 23:02:22.518889 30044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 23:02:22.530167 30044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76d4ae1889544a16a83819222754a5f9: Bootstrap starting.
Aug 07 23:02:22 dist-test-slave-z8x9 krb5kdc[29918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143741, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.62@KRBTEST.COM
I20260807 23:02:22.534240 30044 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 76d4ae1889544a16a83819222754a5f9: Neither blocks nor log segments found. Creating new log.
I20260807 23:02:22.535810 30044 log.cc:824] T 00000000000000000000000000000000 P 76d4ae1889544a16a83819222754a5f9: Log is configured to *not* fsync() on all Append() calls
I20260807 23:02:22.539575 30044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76d4ae1889544a16a83819222754a5f9: No bootstrap required, opened a new log
I20260807 23:02:22.540515 30049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:22.518576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45297 (local address 127.25.94.62:44841)
0807 23:02:22.519428 (+ 852us) server_negotiation.cc:206] Beginning negotiation
0807 23:02:22.519443 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0807 23:02:22.519484 (+ 41us) server_negotiation.cc:407] Connection header received
0807 23:02:22.521261 (+ 1777us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:02:22.521286 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:02:22.521706 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:02:22.522145 (+ 439us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:02:22.523669 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:22.525067 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:22.526387 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:22.527136 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:22.527347 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:02:22.532164 (+ 4817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:02:22.532174 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:02:22.532190 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 23:02:22.532243 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:02:22.535678 (+ 3435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:02:22.536670 (+ 992us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:02:22.536685 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:02:22.536699 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:02:22.536948 (+ 249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:02:22.537466 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:02:22.537477 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:02:22.537480 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:02:22.538026 (+ 546us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:02:22.538247 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0807 23:02:22.538899 (+ 652us) server_negotiation.cc:299] Negotiation successful
0807 23:02:22.539417 (+ 518us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":646,"thread_start_us":415,"threads_started":1}
I20260807 23:02:22.545030 25976 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 23:02:22.549549 30044 raft_consensus.cc:348] T 00000000000000000000000000000000 P 76d4ae1889544a16a83819222754a5f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76d4ae1889544a16a83819222754a5f9" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 44841 } }
I20260807 23:02:22.550028 30044 raft_consensus.cc:374] T 00000000000000000000000000000000 P 76d4ae1889544a16a83819222754a5f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 23:02:22.550141 30044 raft_consensus.cc:729] T 00000000000000000000000000000000 P 76d4ae1889544a16a83819222754a5f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76d4ae1889544a16a83819222754a5f9, State: Initialized, Role: FOLLOWER
I20260807 23:02:22.550890 30044 consensus_queue.cc:260] T 00000000000000000000000000000000 P 76d4ae1889544a16a83819222754a5f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76d4ae1889544a16a83819222754a5f9" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 44841 } }
I20260807 23:02:22.551191 30044 raft_consensus.cc:388] T 00000000000000000000000000000000 P 76d4ae1889544a16a83819222754a5f9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 23:02:22.551369 30044 raft_consensus.cc:482] T 00000000000000000000000000000000 P 76d4ae1889544a16a83819222754a5f9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 23:02:22.551594 30044 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 76d4ae1889544a16a83819222754a5f9 [term 0 FOLLOWER]: Advancing to term 1
I20260807 23:02:22.554759 30044 raft_consensus.cc:504] T 00000000000000000000000000000000 P 76d4ae1889544a16a83819222754a5f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76d4ae1889544a16a83819222754a5f9" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 44841 } }
I20260807 23:02:22.555377 30044 leader_election.cc:302] T 00000000000000000000000000000000 P 76d4ae1889544a16a83819222754a5f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 76d4ae1889544a16a83819222754a5f9; no voters:
I20260807 23:02:22.556021 30044 leader_election.cc:288] T 00000000000000000000000000000000 P 76d4ae1889544a16a83819222754a5f9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 23:02:22.556124 30058 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 76d4ae1889544a16a83819222754a5f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 23:02:22.557401 30058 raft_consensus.cc:686] T 00000000000000000000000000000000 P 76d4ae1889544a16a83819222754a5f9 [term 1 LEADER]: Becoming Leader. State: Replica: 76d4ae1889544a16a83819222754a5f9, State: Running, Role: LEADER
I20260807 23:02:22.558060 30058 consensus_queue.cc:237] T 00000000000000000000000000000000 P 76d4ae1889544a16a83819222754a5f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76d4ae1889544a16a83819222754a5f9" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 44841 } }
I20260807 23:02:22.558483 30044 sys_catalog.cc:564] T 00000000000000000000000000000000 P 76d4ae1889544a16a83819222754a5f9 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 23:02:22.561439 30059 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76d4ae1889544a16a83819222754a5f9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "76d4ae1889544a16a83819222754a5f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76d4ae1889544a16a83819222754a5f9" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 44841 } } }
I20260807 23:02:22.561800 30059 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76d4ae1889544a16a83819222754a5f9 [sys.catalog]: This master's current role is: LEADER
I20260807 23:02:22.561115 30060 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76d4ae1889544a16a83819222754a5f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 76d4ae1889544a16a83819222754a5f9. Latest consensus state: current_term: 1 leader_uuid: "76d4ae1889544a16a83819222754a5f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76d4ae1889544a16a83819222754a5f9" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 44841 } } }
I20260807 23:02:22.562427 30060 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76d4ae1889544a16a83819222754a5f9 [sys.catalog]: This master's current role is: LEADER
I20260807 23:02:22.570691 30049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:22.550581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54307 (local address 127.25.94.62:44841)
0807 23:02:22.550822 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0807 23:02:22.550831 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 23:02:22.551329 (+ 498us) server_negotiation.cc:407] Connection header received
0807 23:02:22.551516 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:02:22.551521 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:02:22.551606 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:02:22.551703 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:02:22.553712 (+ 2009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:22.554433 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:22.556230 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:22.557218 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:22.557383 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:02:22.560875 (+ 3492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:02:22.560885 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:02:22.560892 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 23:02:22.560939 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:02:22.565547 (+ 4608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:02:22.566722 (+ 1175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:02:22.566731 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:02:22.566737 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:02:22.566972 (+ 235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:02:22.567536 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:02:22.567543 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:02:22.567550 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:02:22.568018 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:02:22.569604 (+ 1586us) server_negotiation.cc:1035] Waiting for connection context
0807 23:02:22.569799 (+ 195us) server_negotiation.cc:299] Negotiation successful
0807 23:02:22.570119 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260807 23:02:22.573729 30067 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 23:02:22.580468 30067 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 23:02:22.589789 30067 catalog_manager.cc:1347] Generated new cluster ID: 1e2f73827595456393c0bef885edc2d7
I20260807 23:02:22.589929 30067 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 23:02:22.605350 30067 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 23:02:22.606837 30067 catalog_manager.cc:1504] Loading token signing keys...
I20260807 23:02:22.616106 30067 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 76d4ae1889544a16a83819222754a5f9: Generated new TSK 0
I20260807 23:02:22.617029 30067 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 23:02:22.631136 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 29943
I20260807 23:02:22.632340 30037 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 23:02:22.728781 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 29943
2026-08-07T23:02:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2420 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786143361779964-25976-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 23:02:22 dist-test-slave-z8x9 krb5kdc[30088](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 23:02:22 dist-test-slave-z8x9 krb5kdc[30088](info): set up 2 sockets
Aug 07 23:02:22 dist-test-slave-z8x9 krb5kdc[30088](info): commencing operation
krb5kdc: starting...
W20260807 23:02:24.116468 25976 mini_kdc.cc:121] Time spent starting KDC: real 1.344s user 0.005s sys 0.041s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786143361779964-25976-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786143361779964-25976-0/krb5kdc/test-user.keytab.
Aug 07 23:02:24 dist-test-slave-z8x9 krb5kdc[30088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143744, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T23:02:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T23:02:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.94.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.62.keytab.
Entry for principal kudu/127.25.94.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.62.keytab.
I20260807 23:02:24.470613 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786143361779964-25976-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786143361779964-25976-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786143361779964-25976-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.62:46659
--webserver_interface=127.25.94.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.62.keytab
--principal=kudu/127.25.94.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.20:41839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.94.62:46659
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786143361779964-25976-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786143361779964-25976-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786143361779964-25976-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 23:02:24.828912 30104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 23:02:24.829305 30104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 23:02:24.829423 30104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 23:02:24.836869 30104 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 23:02:24.837014 30104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 23:02:24.837067 30104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 23:02:24.837124 30104 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 23:02:24.837255 30104 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 23:02:24.847525 30104 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:41839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786143361779964-25976-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786143361779964-25976-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.94.62:46659
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.62.keytab
--principal=kudu/127.25.94.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.62:46659
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.94.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786143361779964-25976-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 23:02:24.848884 30104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 23:02:24.850206 30104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 23:02:24.862922 30111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 23:02:24.862914 30114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 23:02:24.863155 30112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 23:02:24.864157 30104 server_base.cc:1034] running on GCE node
I20260807 23:02:24.865001 30104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 23:02:24.866549 30104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 23:02:24.867806 30104 hybrid_clock.cc:648] HybridClock initialized: now 1786143744867746 us; error 44 us; skew 500 ppm
I20260807 23:02:24.868202 30104 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 23:02:24 dist-test-slave-z8x9 krb5kdc[30088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143744, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 23:02:24.878420 30104 init.cc:377] Logged in from keytab as kudu/127.25.94.62@KRBTEST.COM (short username kudu)
I20260807 23:02:24.880111 30104 webserver.cc:460] Webserver started at http://127.25.94.62:43701/ using document root <none> and password file <none>
I20260807 23:02:24.880856 30104 fs_manager.cc:359] Metadata directory not provided
I20260807 23:02:24.881018 30104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 23:02:24.881409 30104 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 23:02:24.884521 30104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786143361779964-25976-0/minicluster-data/master-0/data/instance:
uuid: "f50c6d8db61c4863867f71ebf8338507"
format_stamp: "Formatted at 2026-08-07 23:02:24 on dist-test-slave-z8x9"
server_key: "58bb5f13f846c8961495e9c55b16786b"
server_key_iv: "45cd12668de3222b39e49536cad45b2d"
server_key_version: "encryptionkey@0"
I20260807 23:02:24.885378 30104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786143361779964-25976-0/minicluster-data/master-0/wal/instance:
uuid: "f50c6d8db61c4863867f71ebf8338507"
format_stamp: "Formatted at 2026-08-07 23:02:24 on dist-test-slave-z8x9"
server_key: "58bb5f13f846c8961495e9c55b16786b"
server_key_iv: "45cd12668de3222b39e49536cad45b2d"
server_key_version: "encryptionkey@0"
I20260807 23:02:24.890599 30104 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260807 23:02:24.894145 30124 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 23:02:24.895427 30104 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260807 23:02:24.895625 30104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786143361779964-25976-0/minicluster-data/master-0/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786143361779964-25976-0/minicluster-data/master-0/wal
uuid: "f50c6d8db61c4863867f71ebf8338507"
format_stamp: "Formatted at 2026-08-07 23:02:24 on dist-test-slave-z8x9"
server_key: "58bb5f13f846c8961495e9c55b16786b"
server_key_iv: "45cd12668de3222b39e49536cad45b2d"
server_key_version: "encryptionkey@0"
I20260807 23:02:24.895834 30104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786143361779964-25976-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786143361779964-25976-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786143361779964-25976-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 23:02:24.910561 30104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 23:02:24.915365 30104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 23:02:24.915683 30104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 23:02:24.936659 30104 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.62:46659
I20260807 23:02:24.936694 30200 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.62:46659 every 8 connection(s)
I20260807 23:02:24.938181 30104 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
I20260807 23:02:24.942176 30201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 23:02:24.944296 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 30104
I20260807 23:02:24.944489 25976 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786143361779964-25976-0/minicluster-data/master-0/wal/instance
I20260807 23:02:24.944954 25976 external_mini_cluster.cc:1428] Setting key 72917539d26ce2bc3ebfc3ef713c5241
I20260807 23:02:24.955713 30201 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f50c6d8db61c4863867f71ebf8338507: Bootstrap starting.
I20260807 23:02:24.959539 30201 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f50c6d8db61c4863867f71ebf8338507: Neither blocks nor log segments found. Creating new log.
I20260807 23:02:24.960345 30204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:24.946976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55619 (local address 127.25.94.62:46659)
0807 23:02:24.947851 (+ 875us) server_negotiation.cc:206] Beginning negotiation
0807 23:02:24.947862 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 23:02:24.947911 (+ 49us) server_negotiation.cc:407] Connection header received
0807 23:02:24.949746 (+ 1835us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:02:24.949770 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:02:24.950424 (+ 654us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:02:24.950900 (+ 476us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:02:24.951462 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:24.953762 (+ 2300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:24.955675 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:24.956388 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:24.956528 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:02:24.957912 (+ 1384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:02:24.957919 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:02:24.957932 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 23:02:24.958469 (+ 537us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:02:24.958557 (+ 88us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 23:02:24.958585 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:02:24.958681 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0807 23:02:24.958927 (+ 246us) server_negotiation.cc:299] Negotiation successful
0807 23:02:24.959205 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":640,"thread_start_us":397,"threads_started":1}
I20260807 23:02:24.960743 30201 log.cc:824] T 00000000000000000000000000000000 P f50c6d8db61c4863867f71ebf8338507: Log is configured to *not* fsync() on all Append() calls
I20260807 23:02:24.964185 30201 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f50c6d8db61c4863867f71ebf8338507: No bootstrap required, opened a new log
I20260807 23:02:24.964672 25976 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 23:02:24.973114 30201 raft_consensus.cc:348] T 00000000000000000000000000000000 P f50c6d8db61c4863867f71ebf8338507 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f50c6d8db61c4863867f71ebf8338507" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 46659 } }
I20260807 23:02:24.973636 30201 raft_consensus.cc:374] T 00000000000000000000000000000000 P f50c6d8db61c4863867f71ebf8338507 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 23:02:24.973830 30201 raft_consensus.cc:729] T 00000000000000000000000000000000 P f50c6d8db61c4863867f71ebf8338507 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f50c6d8db61c4863867f71ebf8338507, State: Initialized, Role: FOLLOWER
I20260807 23:02:24.974591 30201 consensus_queue.cc:260] T 00000000000000000000000000000000 P f50c6d8db61c4863867f71ebf8338507 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f50c6d8db61c4863867f71ebf8338507" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 46659 } }
I20260807 23:02:24.974910 30201 raft_consensus.cc:388] T 00000000000000000000000000000000 P f50c6d8db61c4863867f71ebf8338507 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 23:02:24.975047 30201 raft_consensus.cc:482] T 00000000000000000000000000000000 P f50c6d8db61c4863867f71ebf8338507 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 23:02:24.975215 30201 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f50c6d8db61c4863867f71ebf8338507 [term 0 FOLLOWER]: Advancing to term 1
I20260807 23:02:24.976347 30204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:24.969201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59155 (local address 127.25.94.62:46659)
0807 23:02:24.969424 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0807 23:02:24.969433 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 23:02:24.969997 (+ 564us) server_negotiation.cc:407] Connection header received
0807 23:02:24.970174 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:02:24.970178 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:02:24.970251 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:02:24.970336 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:02:24.970975 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:24.971807 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:24.973768 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:24.974283 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:24.974380 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:02:24.975568 (+ 1188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:02:24.975573 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:02:24.975576 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 23:02:24.975593 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:02:24.975637 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 23:02:24.975653 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:02:24.975734 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0807 23:02:24.975914 (+ 180us) server_negotiation.cc:299] Negotiation successful
0807 23:02:24.975988 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260807 23:02:24.978415 30201 raft_consensus.cc:504] T 00000000000000000000000000000000 P f50c6d8db61c4863867f71ebf8338507 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f50c6d8db61c4863867f71ebf8338507" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 46659 } }
I20260807 23:02:24.978950 30201 leader_election.cc:302] T 00000000000000000000000000000000 P f50c6d8db61c4863867f71ebf8338507 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f50c6d8db61c4863867f71ebf8338507; no voters:
I20260807 23:02:24.979614 30201 leader_election.cc:288] T 00000000000000000000000000000000 P f50c6d8db61c4863867f71ebf8338507 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 23:02:24.979779 30214 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f50c6d8db61c4863867f71ebf8338507 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 23:02:24.981213 30214 raft_consensus.cc:686] T 00000000000000000000000000000000 P f50c6d8db61c4863867f71ebf8338507 [term 1 LEADER]: Becoming Leader. State: Replica: f50c6d8db61c4863867f71ebf8338507, State: Running, Role: LEADER
I20260807 23:02:24.981927 30214 consensus_queue.cc:237] T 00000000000000000000000000000000 P f50c6d8db61c4863867f71ebf8338507 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f50c6d8db61c4863867f71ebf8338507" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 46659 } }
I20260807 23:02:24.982143 30201 sys_catalog.cc:564] T 00000000000000000000000000000000 P f50c6d8db61c4863867f71ebf8338507 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 23:02:24.984760 30215 sys_catalog.cc:455] T 00000000000000000000000000000000 P f50c6d8db61c4863867f71ebf8338507 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f50c6d8db61c4863867f71ebf8338507" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f50c6d8db61c4863867f71ebf8338507" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 46659 } } }
I20260807 23:02:24.985265 30215 sys_catalog.cc:458] T 00000000000000000000000000000000 P f50c6d8db61c4863867f71ebf8338507 [sys.catalog]: This master's current role is: LEADER
I20260807 23:02:24.985420 30216 sys_catalog.cc:455] T 00000000000000000000000000000000 P f50c6d8db61c4863867f71ebf8338507 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f50c6d8db61c4863867f71ebf8338507. Latest consensus state: current_term: 1 leader_uuid: "f50c6d8db61c4863867f71ebf8338507" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f50c6d8db61c4863867f71ebf8338507" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 46659 } } }
I20260807 23:02:24.985744 30216 sys_catalog.cc:458] T 00000000000000000000000000000000 P f50c6d8db61c4863867f71ebf8338507 [sys.catalog]: This master's current role is: LEADER
W20260807 23:02:24.992729 30235 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f50c6d8db61c4863867f71ebf8338507: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 23:02:24.992879 30235 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 23:02:24.993042 30228 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 23:02:24.994771 30228 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 23:02:25.004172 30228 catalog_manager.cc:1347] Generated new cluster ID: 93dfb2fe4c1f4ed1a55ec7409a8d6b38
I20260807 23:02:25.004307 30228 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 23:02:25.021829 30228 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 23:02:25.023227 30228 catalog_manager.cc:1504] Loading token signing keys...
I20260807 23:02:25.035703 30228 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f50c6d8db61c4863867f71ebf8338507: Generated new TSK 0
I20260807 23:02:25.036854 30228 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 23:02:25.051733 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 30104
W20260807 23:02:25.052843 30189 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:55619
W20260807 23:02:25.053603 25976 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 30104: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 23:02:25.053759 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 30104
2026-08-07T23:02:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (2322 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786143361779964-25976-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 23:02:25 dist-test-slave-z8x9 krb5kdc[30243](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 23:02:25 dist-test-slave-z8x9 krb5kdc[30243](info): set up 2 sockets
Aug 07 23:02:25 dist-test-slave-z8x9 krb5kdc[30243](info): commencing operation
krb5kdc: starting...
W20260807 23:02:26.450930 25976 mini_kdc.cc:121] Time spent starting KDC: real 1.356s user 0.004s sys 0.048s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786143361779964-25976-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786143361779964-25976-0/krb5kdc/test-user.keytab.
Aug 07 23:02:26 dist-test-slave-z8x9 krb5kdc[30243](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143746, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T23:02:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T23:02:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.94.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.62.keytab.
Entry for principal kudu/127.25.94.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.62.keytab.
I20260807 23:02:26.800884 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786143361779964-25976-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786143361779964-25976-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786143361779964-25976-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.62:38947
--webserver_interface=127.25.94.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.62.keytab
--principal=kudu/127.25.94.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.20:35129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.94.62:38947
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786143361779964-25976-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786143361779964-25976-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786143361779964-25976-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 23:02:27.155550 30269 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 23:02:27.155891 30269 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 23:02:27.156082 30269 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 23:02:27.163654 30269 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 23:02:27.163798 30269 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 23:02:27.163895 30269 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 23:02:27.163993 30269 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 23:02:27.164088 30269 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 23:02:27.174584 30269 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:35129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786143361779964-25976-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786143361779964-25976-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.94.62:38947
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.62.keytab
--principal=kudu/127.25.94.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.62:38947
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.94.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786143361779964-25976-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 23:02:27.176090 30269 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 23:02:27.177556 30269 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 23:02:27.190999 30280 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 23:02:27.191064 30282 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 23:02:27.191093 30278 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 23:02:27.192040 30269 server_base.cc:1034] running on GCE node
I20260807 23:02:27.192983 30269 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 23:02:27.194650 30269 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 23:02:27.195932 30269 hybrid_clock.cc:648] HybridClock initialized: now 1786143747195890 us; error 36 us; skew 500 ppm
I20260807 23:02:27.196331 30269 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 23:02:27 dist-test-slave-z8x9 krb5kdc[30243](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143747, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 23:02:27.199894 30269 init.cc:377] Logged in from keytab as kudu/127.25.94.62@KRBTEST.COM (short username kudu)
I20260807 23:02:27.201651 30269 webserver.cc:460] Webserver started at http://127.25.94.62:44445/ using document root <none> and password file <none>
I20260807 23:02:27.202438 30269 fs_manager.cc:359] Metadata directory not provided
I20260807 23:02:27.202617 30269 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 23:02:27.202962 30269 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 23:02:27.206351 30269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786143361779964-25976-0/minicluster-data/master-0/data/instance:
uuid: "fdfdc70cba1f49c5bc4d0170238c972b"
format_stamp: "Formatted at 2026-08-07 23:02:27 on dist-test-slave-z8x9"
server_key: "a70cc9f84db5c465d175076db283905e"
server_key_iv: "9a84f06b83fdead676378c56b44794f8"
server_key_version: "encryptionkey@0"
I20260807 23:02:27.207432 30269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786143361779964-25976-0/minicluster-data/master-0/wal/instance:
uuid: "fdfdc70cba1f49c5bc4d0170238c972b"
format_stamp: "Formatted at 2026-08-07 23:02:27 on dist-test-slave-z8x9"
server_key: "a70cc9f84db5c465d175076db283905e"
server_key_iv: "9a84f06b83fdead676378c56b44794f8"
server_key_version: "encryptionkey@0"
I20260807 23:02:27.213472 30269 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260807 23:02:27.217367 30290 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 23:02:27.218917 30269 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 23:02:27.219153 30269 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786143361779964-25976-0/minicluster-data/master-0/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786143361779964-25976-0/minicluster-data/master-0/wal
uuid: "fdfdc70cba1f49c5bc4d0170238c972b"
format_stamp: "Formatted at 2026-08-07 23:02:27 on dist-test-slave-z8x9"
server_key: "a70cc9f84db5c465d175076db283905e"
server_key_iv: "9a84f06b83fdead676378c56b44794f8"
server_key_version: "encryptionkey@0"
I20260807 23:02:27.219404 30269 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786143361779964-25976-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786143361779964-25976-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786143361779964-25976-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 23:02:27.225639 30269 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 23:02:27.230389 30269 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 23:02:27.230731 30269 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 23:02:27.251096 30269 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.62:38947
I20260807 23:02:27.251165 30371 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.62:38947 every 8 connection(s)
I20260807 23:02:27.252616 30269 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
I20260807 23:02:27.255424 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 30269
I20260807 23:02:27.255649 25976 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786143361779964-25976-0/minicluster-data/master-0/wal/instance
I20260807 23:02:27.256107 25976 external_mini_cluster.cc:1428] Setting key 8d26e3d2679fee4ffb5f2d4798a9ba74
I20260807 23:02:27.256924 30372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 23:02:27.266125 30375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:27.258151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56283 (local address 127.25.94.62:38947)
0807 23:02:27.258990 (+ 839us) server_negotiation.cc:206] Beginning negotiation
0807 23:02:27.259004 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0807 23:02:27.259064 (+ 60us) server_negotiation.cc:407] Connection header received
0807 23:02:27.261960 (+ 2896us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:02:27.261995 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:02:27.262511 (+ 516us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:02:27.262999 (+ 488us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:02:27.264071 (+ 1072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:02:27.264080 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:02:27.264103 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 23:02:27.264126 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:02:27.264192 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 23:02:27.264220 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:02:27.264492 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0807 23:02:27.264626 (+ 134us) server_negotiation.cc:299] Negotiation successful
0807 23:02:27.264841 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":586,"thread_start_us":352,"threads_started":1}
I20260807 23:02:27.270154 25976 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 23:02:27.271831 30372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fdfdc70cba1f49c5bc4d0170238c972b: Bootstrap starting.
I20260807 23:02:27.275655 30372 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fdfdc70cba1f49c5bc4d0170238c972b: Neither blocks nor log segments found. Creating new log.
I20260807 23:02:27.276820 30372 log.cc:824] T 00000000000000000000000000000000 P fdfdc70cba1f49c5bc4d0170238c972b: Log is configured to *not* fsync() on all Append() calls
I20260807 23:02:27.277977 30375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:27.275048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33001 (local address 127.25.94.62:38947)
0807 23:02:27.275299 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0807 23:02:27.275308 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 23:02:27.275649 (+ 341us) server_negotiation.cc:407] Connection header received
0807 23:02:27.275914 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:02:27.275918 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:02:27.275969 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:02:27.276106 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:02:27.277110 (+ 1004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:02:27.277117 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:02:27.277122 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 23:02:27.277141 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:02:27.277224 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 23:02:27.277248 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:02:27.277397 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0807 23:02:27.277554 (+ 157us) server_negotiation.cc:299] Negotiation successful
0807 23:02:27.277655 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260807 23:02:27.280462 30372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fdfdc70cba1f49c5bc4d0170238c972b: No bootstrap required, opened a new log
I20260807 23:02:27.288116 30372 raft_consensus.cc:348] T 00000000000000000000000000000000 P fdfdc70cba1f49c5bc4d0170238c972b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdfdc70cba1f49c5bc4d0170238c972b" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 38947 } }
I20260807 23:02:27.288568 30372 raft_consensus.cc:374] T 00000000000000000000000000000000 P fdfdc70cba1f49c5bc4d0170238c972b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 23:02:27.288728 30372 raft_consensus.cc:729] T 00000000000000000000000000000000 P fdfdc70cba1f49c5bc4d0170238c972b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdfdc70cba1f49c5bc4d0170238c972b, State: Initialized, Role: FOLLOWER
I20260807 23:02:27.289521 30372 consensus_queue.cc:260] T 00000000000000000000000000000000 P fdfdc70cba1f49c5bc4d0170238c972b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdfdc70cba1f49c5bc4d0170238c972b" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 38947 } }
I20260807 23:02:27.289805 30372 raft_consensus.cc:388] T 00000000000000000000000000000000 P fdfdc70cba1f49c5bc4d0170238c972b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 23:02:27.289978 30372 raft_consensus.cc:482] T 00000000000000000000000000000000 P fdfdc70cba1f49c5bc4d0170238c972b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 23:02:27.290164 30372 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fdfdc70cba1f49c5bc4d0170238c972b [term 0 FOLLOWER]: Advancing to term 1
I20260807 23:02:27.292466 30372 raft_consensus.cc:504] T 00000000000000000000000000000000 P fdfdc70cba1f49c5bc4d0170238c972b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdfdc70cba1f49c5bc4d0170238c972b" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 38947 } }
I20260807 23:02:27.293021 30372 leader_election.cc:302] T 00000000000000000000000000000000 P fdfdc70cba1f49c5bc4d0170238c972b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fdfdc70cba1f49c5bc4d0170238c972b; no voters:
I20260807 23:02:27.293715 30372 leader_election.cc:288] T 00000000000000000000000000000000 P fdfdc70cba1f49c5bc4d0170238c972b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 23:02:27.293882 30384 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fdfdc70cba1f49c5bc4d0170238c972b [term 1 FOLLOWER]: Leader election won for term 1
I20260807 23:02:27.295189 30384 raft_consensus.cc:686] T 00000000000000000000000000000000 P fdfdc70cba1f49c5bc4d0170238c972b [term 1 LEADER]: Becoming Leader. State: Replica: fdfdc70cba1f49c5bc4d0170238c972b, State: Running, Role: LEADER
I20260807 23:02:27.295851 30384 consensus_queue.cc:237] T 00000000000000000000000000000000 P fdfdc70cba1f49c5bc4d0170238c972b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdfdc70cba1f49c5bc4d0170238c972b" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 38947 } }
I20260807 23:02:27.296121 30372 sys_catalog.cc:564] T 00000000000000000000000000000000 P fdfdc70cba1f49c5bc4d0170238c972b [sys.catalog]: configured and running, proceeding with master startup.
I20260807 23:02:27.299036 30390 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdfdc70cba1f49c5bc4d0170238c972b [sys.catalog]: SysCatalogTable state changed. Reason: New leader fdfdc70cba1f49c5bc4d0170238c972b. Latest consensus state: current_term: 1 leader_uuid: "fdfdc70cba1f49c5bc4d0170238c972b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdfdc70cba1f49c5bc4d0170238c972b" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 38947 } } }
I20260807 23:02:27.299579 30390 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdfdc70cba1f49c5bc4d0170238c972b [sys.catalog]: This master's current role is: LEADER
I20260807 23:02:27.299510 30387 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdfdc70cba1f49c5bc4d0170238c972b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fdfdc70cba1f49c5bc4d0170238c972b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdfdc70cba1f49c5bc4d0170238c972b" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 38947 } } }
I20260807 23:02:27.299785 30387 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdfdc70cba1f49c5bc4d0170238c972b [sys.catalog]: This master's current role is: LEADER
I20260807 23:02:27.300560 30401 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 23:02:27.306396 30401 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 23:02:27.315662 30401 catalog_manager.cc:1347] Generated new cluster ID: c6fd71b241d642a2b7516c7ceb934a15
I20260807 23:02:27.315806 30401 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 23:02:27.342976 30401 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 23:02:27.343518 30401 catalog_manager.cc:1504] Loading token signing keys...
I20260807 23:02:27.353328 30401 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fdfdc70cba1f49c5bc4d0170238c972b: Generated new TSK 0
I20260807 23:02:27.354219 30401 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 23:02:27.371935 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 30269
W20260807 23:02:27.373108 30365 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:56283
W20260807 23:02:27.373930 25976 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 30269: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 23:02:27.374099 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 30269
2026-08-07T23:02:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (2333 ms)
[----------] 18 tests from AuthTokenIssuingTest (42307 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786143361779964-25976-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 23:02:27 dist-test-slave-z8x9 krb5kdc[30416](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 23:02:27 dist-test-slave-z8x9 krb5kdc[30416](info): set up 2 sockets
Aug 07 23:02:27 dist-test-slave-z8x9 krb5kdc[30416](info): commencing operation
krb5kdc: starting...
W20260807 23:02:28.765803 25976 mini_kdc.cc:121] Time spent starting KDC: real 1.336s user 0.007s sys 0.041s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786143361779964-25976-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786143361779964-25976-0/krb5kdc/test-user.keytab.
Aug 07 23:02:28 dist-test-slave-z8x9 krb5kdc[30416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143748, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T23:02:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T23:02:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.94.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.62.keytab.
Entry for principal kudu/127.25.94.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.62.keytab.
I20260807 23:02:29.123538 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786143361779964-25976-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786143361779964-25976-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786143361779964-25976-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.62:34799
--webserver_interface=127.25.94.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.62.keytab
--principal=kudu/127.25.94.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.20:45191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.94.62:34799,127.25.94.61:38653,127.25.94.60:37557
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786143361779964-25976-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786143361779964-25976-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786143361779964-25976-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 23:02:29.479307 30436 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260807 23:02:29.479664 30436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 23:02:29.479828 30436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 23:02:29.479956 30436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 23:02:29.487349 30436 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 23:02:29.487457 30436 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 23:02:29.487550 30436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 23:02:29.487648 30436 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 23:02:29.487756 30436 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 23:02:29.497800 30436 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:45191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786143361779964-25976-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786143361779964-25976-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.94.62:34799,127.25.94.61:38653,127.25.94.60:37557
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.62.keytab
--principal=kudu/127.25.94.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.62:34799
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.94.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786143361779964-25976-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 23:02:29.499135 30436 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 23:02:29.500450 30436 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 23:02:29.512564 30444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 23:02:29.512655 30446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 23:02:29.512571 30443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 23:02:29.513429 30436 server_base.cc:1034] running on GCE node
I20260807 23:02:29.514246 30436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 23:02:29.515779 30436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 23:02:29.517071 30436 hybrid_clock.cc:648] HybridClock initialized: now 1786143749517007 us; error 36 us; skew 500 ppm
I20260807 23:02:29.517526 30436 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 23:02:29 dist-test-slave-z8x9 krb5kdc[30416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143749, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 23:02:29.521108 30436 init.cc:377] Logged in from keytab as kudu/127.25.94.62@KRBTEST.COM (short username kudu)
I20260807 23:02:29.522626 30436 webserver.cc:460] Webserver started at http://127.25.94.62:40993/ using document root <none> and password file <none>
I20260807 23:02:29.523347 30436 fs_manager.cc:359] Metadata directory not provided
I20260807 23:02:29.523438 30436 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 23:02:29.523699 30436 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 23:02:29.526726 30436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786143361779964-25976-0/minicluster-data/master-0/data/instance:
uuid: "1a6b04948f9c45ce99e9a420c6b2563b"
format_stamp: "Formatted at 2026-08-07 23:02:29 on dist-test-slave-z8x9"
server_key: "3009057bfb5d56e3c82fabc79f671247"
server_key_iv: "c64197d15ce719114c97e9f31e7ee238"
server_key_version: "encryptionkey@0"
I20260807 23:02:29.527465 30436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786143361779964-25976-0/minicluster-data/master-0/wal/instance:
uuid: "1a6b04948f9c45ce99e9a420c6b2563b"
format_stamp: "Formatted at 2026-08-07 23:02:29 on dist-test-slave-z8x9"
server_key: "3009057bfb5d56e3c82fabc79f671247"
server_key_iv: "c64197d15ce719114c97e9f31e7ee238"
server_key_version: "encryptionkey@0"
I20260807 23:02:29.532584 30436 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260807 23:02:29.536017 30465 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 23:02:29.537611 30436 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260807 23:02:29.537806 30436 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786143361779964-25976-0/minicluster-data/master-0/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786143361779964-25976-0/minicluster-data/master-0/wal
uuid: "1a6b04948f9c45ce99e9a420c6b2563b"
format_stamp: "Formatted at 2026-08-07 23:02:29 on dist-test-slave-z8x9"
server_key: "3009057bfb5d56e3c82fabc79f671247"
server_key_iv: "c64197d15ce719114c97e9f31e7ee238"
server_key_version: "encryptionkey@0"
I20260807 23:02:29.538022 30436 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786143361779964-25976-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786143361779964-25976-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786143361779964-25976-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 23:02:29.553866 30436 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 23:02:29.559149 30436 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 23:02:29.559459 30436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 23:02:29.580472 30436 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.62:34799
I20260807 23:02:29.580518 30550 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.62:34799 every 8 connection(s)
I20260807 23:02:29.581964 30436 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
I20260807 23:02:29.586426 30552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 23:02:29.589941 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 30436
I20260807 23:02:29.590170 25976 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786143361779964-25976-0/minicluster-data/master-0/wal/instance
I20260807 23:02:29.591010 25976 external_mini_cluster.cc:1428] Setting key 1a232f51d1777cc9e20581edb54d386d
I20260807 23:02:29.595950 30552 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.94.62" port: 34799 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 23:02:29 dist-test-slave-z8x9 krb5kdc[30416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143749, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.62@KRBTEST.COM for kudu/127.25.94.62@KRBTEST.COM
I20260807 23:02:29.622308 30556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:29.600408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32958 (local address 127.25.94.62:34799)
0807 23:02:29.601355 (+ 947us) server_negotiation.cc:206] Beginning negotiation
0807 23:02:29.601372 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0807 23:02:29.601970 (+ 598us) server_negotiation.cc:407] Connection header received
0807 23:02:29.604047 (+ 2077us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:02:29.604057 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:02:29.604367 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:02:29.604593 (+ 226us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:02:29.605910 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:29.607207 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:29.608628 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:29.609677 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:29.609907 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:02:29.613938 (+ 4031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:02:29.613946 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:02:29.613961 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 23:02:29.613991 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:02:29.617476 (+ 3485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:02:29.618545 (+ 1069us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:02:29.618553 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:02:29.618556 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:02:29.618754 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:02:29.619251 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:02:29.619255 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:02:29.619257 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:02:29.619818 (+ 561us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:02:29.619984 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0807 23:02:29.620724 (+ 740us) server_negotiation.cc:299] Negotiation successful
0807 23:02:29.621121 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":50,"server-negotiator.queue_time_us":656,"thread_start_us":382,"threads_started":1}
I20260807 23:02:29.622375 30557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:29.600408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:34799 (local address 127.0.0.1:32958)
0807 23:02:29.601775 (+ 1367us) negotiation.cc:107] Waiting for socket to connect
0807 23:02:29.601818 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0807 23:02:29.603635 (+ 1817us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 23:02:29.604640 (+ 1005us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 23:02:29.604645 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 23:02:29.605310 (+ 665us) client_negotiation.cc:189] Negotiated authn=SASL
0807 23:02:29.605663 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:02:29.605679 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:02:29.607377 (+ 1698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:02:29.607382 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:02:29.608474 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:02:29.608487 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:02:29.609937 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:02:29.609942 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:02:29.610125 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:02:29.610904 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 23:02:29.610928 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0807 23:02:29.613687 (+ 2759us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 23:02:29.617641 (+ 3954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 23:02:29.617658 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 23:02:29.617674 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0807 23:02:29.618334 (+ 660us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 23:02:29.618894 (+ 560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 23:02:29.618902 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 23:02:29.618908 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 23:02:29.619133 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 23:02:29.619976 (+ 843us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 23:02:29.619989 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 23:02:29.620305 (+ 316us) client_negotiation.cc:769] Sending connection context
0807 23:02:29.620606 (+ 301us) client_negotiation.cc:240] Negotiation successful
0807 23:02:29.621340 (+ 734us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1091,"mutex_wait_us":165,"thread_start_us":840,"threads_started":1}
WARNING: no policy specified for kudu/127.25.94.61@KRBTEST.COM; defaulting to no policy
I20260807 23:02:29.625973 30552 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.94.61" port: 38653 } has no permanent_uuid. Determining permanent_uuid...
I20260807 23:02:29.627161 30557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:29.626698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.61:38653 (local address 127.0.0.1:35262)
0807 23:02:29.626917 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0807 23:02:29.627042 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.61:38653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
W20260807 23:02:29.627465 30474 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.94.61:38653: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.61@KRBTEST.COM" created.
I20260807 23:02:29.629279 30557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:29.628765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.61:38653 (local address 127.0.0.1:35264)
0807 23:02:29.628954 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0807 23:02:29.629072 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.61:38653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260807 23:02:29.629513 30552 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.94.61:38653: Network error: Client connection negotiation failed: client connection to 127.25.94.61:38653: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.61.keytab.
Entry for principal kudu/127.25.94.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.61.keytab.
I20260807 23:02:29.668906 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786143361779964-25976-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786143361779964-25976-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786143361779964-25976-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786143361779964-25976-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.61:38653
--webserver_interface=127.25.94.61
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.61.keytab
--principal=kudu/127.25.94.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.20:45191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.94.62:34799,127.25.94.61:38653,127.25.94.60:37557
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786143361779964-25976-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786143361779964-25976-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786143361779964-25976-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 23:02:29.678931 30552 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.94.61" port: 38653 } attempt: 1
I20260807 23:02:29.680311 30557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:29.679710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.61:38653 (local address 127.0.0.1:35272)
0807 23:02:29.679962 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0807 23:02:29.680103 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.61:38653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260807 23:02:29.681627 30557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:29.681149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.61:38653 (local address 127.0.0.1:35280)
0807 23:02:29.681397 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0807 23:02:29.681493 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.61:38653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
W20260807 23:02:29.681818 30552 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.94.61:38653: Network error: Client connection negotiation failed: client connection to 127.25.94.61:38653: connect: Connection refused (error 111)
I20260807 23:02:29.750077 30552 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.94.61" port: 38653 } attempt: 2
I20260807 23:02:29.751375 30557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:29.750879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.61:38653 (local address 127.0.0.1:35286)
0807 23:02:29.751129 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0807 23:02:29.751230 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.61:38653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260807 23:02:29.752493 30557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:29.752124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.61:38653 (local address 127.0.0.1:35288)
0807 23:02:29.752300 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0807 23:02:29.752369 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.61:38653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260807 23:02:29.752792 30552 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.94.61:38653: Network error: Client connection negotiation failed: client connection to 127.25.94.61:38653: connect: Connection refused (error 111)
I20260807 23:02:29.844072 30552 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.94.61" port: 38653 } attempt: 3
I20260807 23:02:29.845487 30557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:29.844871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.61:38653 (local address 127.0.0.1:35304)
0807 23:02:29.845123 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0807 23:02:29.845330 (+ 207us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.61:38653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
I20260807 23:02:29.846530 30557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:29.846159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.61:38653 (local address 127.0.0.1:35316)
0807 23:02:29.846316 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0807 23:02:29.846395 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.61:38653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260807 23:02:29.846726 30552 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.94.61:38653: Network error: Client connection negotiation failed: client connection to 127.25.94.61:38653: connect: Connection refused (error 111)
I20260807 23:02:29.989995 30552 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.94.61" port: 38653 } attempt: 4
I20260807 23:02:29.991422 30557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:29.990802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.61:38653 (local address 127.0.0.1:35330)
0807 23:02:29.991157 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0807 23:02:29.991252 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.61:38653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206}
I20260807 23:02:29.992540 30557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:29.992139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.61:38653 (local address 127.0.0.1:35334)
0807 23:02:29.992296 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0807 23:02:29.992402 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.61:38653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260807 23:02:29.992740 30552 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.94.61:38653: Network error: Client connection negotiation failed: client connection to 127.25.94.61:38653: connect: Connection refused (error 111)
W20260807 23:02:30.025576 30564 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260807 23:02:30.026005 30564 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 23:02:30.026134 30564 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 23:02:30.026278 30564 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 23:02:30.033910 30564 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 23:02:30.034053 30564 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 23:02:30.034183 30564 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 23:02:30.034271 30564 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 23:02:30.034387 30564 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 23:02:30.045315 30564 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:45191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786143361779964-25976-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786143361779964-25976-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.25.94.62:34799,127.25.94.61:38653,127.25.94.60:37557
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.61.keytab
--principal=kudu/127.25.94.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.61:38653
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786143361779964-25976-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.25.94.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786143361779964-25976-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 23:02:30.046821 30564 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 23:02:30.048175 30564 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 23:02:30.061061 30570 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 23:02:30.061152 30571 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 23:02:30.061127 30574 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 23:02:30.062062 30564 server_base.cc:1034] running on GCE node
I20260807 23:02:30.062949 30564 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 23:02:30.064435 30564 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 23:02:30.065683 30564 hybrid_clock.cc:648] HybridClock initialized: now 1786143750065604 us; error 61 us; skew 500 ppm
I20260807 23:02:30.066027 30564 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 23:02:30 dist-test-slave-z8x9 krb5kdc[30416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143750, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 23:02:30.074587 30564 init.cc:377] Logged in from keytab as kudu/127.25.94.61@KRBTEST.COM (short username kudu)
I20260807 23:02:30.076293 30564 webserver.cc:460] Webserver started at http://127.25.94.61:35891/ using document root <none> and password file <none>
I20260807 23:02:30.077057 30564 fs_manager.cc:359] Metadata directory not provided
I20260807 23:02:30.077245 30564 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 23:02:30.077596 30564 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 23:02:30.080753 30564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786143361779964-25976-0/minicluster-data/master-1/data/instance:
uuid: "9bcfdd00153040c7ae21f38a2ffaf010"
format_stamp: "Formatted at 2026-08-07 23:02:30 on dist-test-slave-z8x9"
server_key: "4e8d81c2ba8ba1540235f2bcd868d2a5"
server_key_iv: "41bd201dd815738b4c4e6d4a8b2253fa"
server_key_version: "encryptionkey@0"
I20260807 23:02:30.081728 30564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786143361779964-25976-0/minicluster-data/master-1/wal/instance:
uuid: "9bcfdd00153040c7ae21f38a2ffaf010"
format_stamp: "Formatted at 2026-08-07 23:02:30 on dist-test-slave-z8x9"
server_key: "4e8d81c2ba8ba1540235f2bcd868d2a5"
server_key_iv: "41bd201dd815738b4c4e6d4a8b2253fa"
server_key_version: "encryptionkey@0"
I20260807 23:02:30.087229 30564 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260807 23:02:30.091125 30587 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 23:02:30.092870 30564 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260807 23:02:30.093097 30564 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786143361779964-25976-0/minicluster-data/master-1/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786143361779964-25976-0/minicluster-data/master-1/wal
uuid: "9bcfdd00153040c7ae21f38a2ffaf010"
format_stamp: "Formatted at 2026-08-07 23:02:30 on dist-test-slave-z8x9"
server_key: "4e8d81c2ba8ba1540235f2bcd868d2a5"
server_key_iv: "41bd201dd815738b4c4e6d4a8b2253fa"
server_key_version: "encryptionkey@0"
I20260807 23:02:30.093398 30564 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786143361779964-25976-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786143361779964-25976-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786143361779964-25976-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 23:02:30.117790 30564 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 23:02:30.123271 30564 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 23:02:30.123639 30564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 23:02:30.146664 30564 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.61:38653
I20260807 23:02:30.146695 30674 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.61:38653 every 8 connection(s)
I20260807 23:02:30.148196 30564 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786143361779964-25976-0/minicluster-data/master-1/data/info.pb
I20260807 23:02:30.152771 30676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 23:02:30.154057 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 30564
I20260807 23:02:30.154280 25976 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786143361779964-25976-0/minicluster-data/master-1/wal/instance
I20260807 23:02:30.154927 25976 external_mini_cluster.cc:1428] Setting key 64a7abe890a18b7e281fd896f242f88f
I20260807 23:02:30.163604 30676 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.94.62" port: 34799 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 23:02:30 dist-test-slave-z8x9 krb5kdc[30416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143750, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.61@KRBTEST.COM for kudu/127.25.94.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.25.94.60@KRBTEST.COM; defaulting to no policy
I20260807 23:02:30.195600 30679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:30.167795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32960 (local address 127.25.94.62:34799)
0807 23:02:30.168404 (+ 609us) server_negotiation.cc:206] Beginning negotiation
0807 23:02:30.168411 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 23:02:30.169569 (+ 1158us) server_negotiation.cc:407] Connection header received
0807 23:02:30.172123 (+ 2554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:02:30.172132 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:02:30.172241 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:02:30.172414 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:02:30.174555 (+ 2141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:30.175839 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:30.178401 (+ 2562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:30.179245 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:30.179426 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:02:30.185032 (+ 5606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:02:30.185045 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:02:30.185053 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 23:02:30.185149 (+ 96us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:02:30.190162 (+ 5013us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:02:30.191928 (+ 1766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:02:30.191935 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:02:30.191939 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:02:30.192145 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:02:30.192948 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:02:30.192956 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:02:30.192962 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:02:30.193490 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:02:30.193676 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0807 23:02:30.194882 (+ 1206us) server_negotiation.cc:299] Negotiation successful
0807 23:02:30.195146 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":472,"thread_start_us":302,"threads_started":1}
I20260807 23:02:30.196789 30680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:30.168372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:34799 (local address 127.0.0.1:32960)
0807 23:02:30.169351 (+ 979us) negotiation.cc:107] Waiting for socket to connect
0807 23:02:30.169405 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0807 23:02:30.171770 (+ 2365us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 23:02:30.172889 (+ 1119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 23:02:30.172900 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 23:02:30.173620 (+ 720us) client_negotiation.cc:189] Negotiated authn=SASL
0807 23:02:30.174273 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:02:30.174294 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:02:30.176209 (+ 1915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:02:30.176216 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:02:30.178148 (+ 1932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:02:30.178174 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:02:30.179600 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:02:30.179606 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:02:30.179866 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:02:30.180730 (+ 864us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 23:02:30.180758 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0807 23:02:30.184511 (+ 3753us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 23:02:30.190535 (+ 6024us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 23:02:30.190549 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 23:02:30.190573 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0807 23:02:30.191665 (+ 1092us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 23:02:30.192465 (+ 800us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 23:02:30.192471 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 23:02:30.192474 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 23:02:30.192757 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 23:02:30.193692 (+ 935us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 23:02:30.193700 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 23:02:30.194361 (+ 661us) client_negotiation.cc:769] Sending connection context
0807 23:02:30.194761 (+ 400us) client_negotiation.cc:240] Negotiation successful
0807 23:02:30.195399 (+ 638us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614,"thread_start_us":363,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.60@KRBTEST.COM" created.
I20260807 23:02:30.199052 30676 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.94.61" port: 38653 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 23:02:30 dist-test-slave-z8x9 krb5kdc[30416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143750, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.61@KRBTEST.COM for kudu/127.25.94.61@KRBTEST.COM
I20260807 23:02:30.216920 30680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:30.200000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.61:38653 (local address 127.0.0.1:35346)
0807 23:02:30.200173 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0807 23:02:30.200189 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 23:02:30.200320 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 23:02:30.201819 (+ 1499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 23:02:30.201824 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 23:02:30.202321 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0807 23:02:30.202452 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:02:30.202460 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:02:30.203878 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:02:30.203884 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:02:30.204755 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:02:30.204767 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:02:30.205895 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:02:30.205900 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:02:30.206082 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:02:30.206829 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 23:02:30.206850 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0807 23:02:30.209153 (+ 2303us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 23:02:30.214015 (+ 4862us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 23:02:30.214023 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 23:02:30.214030 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0807 23:02:30.214411 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 23:02:30.214813 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 23:02:30.214819 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 23:02:30.214827 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0807 23:02:30.214969 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 23:02:30.215598 (+ 629us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 23:02:30.215606 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 23:02:30.215757 (+ 151us) client_negotiation.cc:769] Sending connection context
0807 23:02:30.215902 (+ 145us) client_negotiation.cc:240] Negotiation successful
0807 23:02:30.216375 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260807 23:02:30.217012 30683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:30.200323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35346 (local address 127.25.94.61:38653)
0807 23:02:30.201330 (+ 1007us) server_negotiation.cc:206] Beginning negotiation
0807 23:02:30.201338 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 23:02:30.201369 (+ 31us) server_negotiation.cc:407] Connection header received
0807 23:02:30.201423 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:02:30.201433 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:02:30.201653 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:02:30.201838 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:02:30.202640 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:30.203741 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:30.204907 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:30.205716 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:30.205886 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:02:30.209552 (+ 3666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:02:30.209572 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:02:30.209592 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 23:02:30.209638 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:02:30.213838 (+ 4200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:02:30.214554 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:02:30.214568 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:02:30.214570 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:02:30.214686 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:02:30.215082 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:02:30.215089 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:02:30.215092 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:02:30.215462 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:02:30.215686 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0807 23:02:30.216078 (+ 392us) server_negotiation.cc:299] Negotiation successful
0807 23:02:30.216543 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":163,"server-negotiator.queue_time_us":714,"thread_start_us":458,"threads_started":1}
I20260807 23:02:30.219590 30676 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.94.60" port: 37557 } has no permanent_uuid. Determining permanent_uuid...
I20260807 23:02:30.220948 30680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:30.220340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.60:37557 (local address 127.0.0.1:45862)
0807 23:02:30.220579 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0807 23:02:30.220737 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.60:37557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260807 23:02:30.221321 30592 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.94.60:37557: connect: Connection refused (error 111)
I20260807 23:02:30.223169 30680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:30.222686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.60:37557 (local address 127.0.0.1:45870)
0807 23:02:30.222883 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0807 23:02:30.223000 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.60:37557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260807 23:02:30.223519 30676 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.94.60:37557: Network error: Client connection negotiation failed: client connection to 127.25.94.60:37557: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.60.keytab.
Entry for principal kudu/127.25.94.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.60.keytab.
I20260807 23:02:30.243417 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786143361779964-25976-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786143361779964-25976-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786143361779964-25976-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786143361779964-25976-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.60:37557
--webserver_interface=127.25.94.60
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.60.keytab
--principal=kudu/127.25.94.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.20:45191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.94.62:34799,127.25.94.61:38653,127.25.94.60:37557
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786143361779964-25976-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786143361779964-25976-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786143361779964-25976-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 23:02:30.273025 30676 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.94.60" port: 37557 } attempt: 1
I20260807 23:02:30.274441 30680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:30.273910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.60:37557 (local address 127.0.0.1:45886)
0807 23:02:30.274144 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0807 23:02:30.274285 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.60:37557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260807 23:02:30.275514 30680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:30.275146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.60:37557 (local address 127.0.0.1:45900)
0807 23:02:30.275321 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0807 23:02:30.275408 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.60:37557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260807 23:02:30.275718 30676 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.94.60:37557: Network error: Client connection negotiation failed: client connection to 127.25.94.60:37557: connect: Connection refused (error 111)
I20260807 23:02:30.292021 30552 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.94.61" port: 38653 } attempt: 5
Aug 07 23:02:30 dist-test-slave-z8x9 krb5kdc[30416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143749, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.62@KRBTEST.COM for kudu/127.25.94.61@KRBTEST.COM
I20260807 23:02:30.306880 30557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:30.292805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.61:38653 (local address 127.0.0.1:35352)
0807 23:02:30.293000 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0807 23:02:30.293022 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 23:02:30.293137 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 23:02:30.293571 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 23:02:30.293574 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 23:02:30.293992 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0807 23:02:30.294173 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:02:30.294182 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:02:30.295520 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:02:30.295524 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:02:30.296362 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:02:30.296373 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:02:30.297434 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:02:30.297439 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:02:30.297582 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:02:30.298366 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 23:02:30.298383 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 23:02:30.300489 (+ 2106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 23:02:30.304505 (+ 4016us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 23:02:30.304509 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 23:02:30.304512 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 23:02:30.304902 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 23:02:30.305343 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 23:02:30.305347 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 23:02:30.305349 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 23:02:30.305478 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 23:02:30.306105 (+ 627us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 23:02:30.306108 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 23:02:30.306249 (+ 141us) client_negotiation.cc:769] Sending connection context
0807 23:02:30.306394 (+ 145us) client_negotiation.cc:240] Negotiation successful
0807 23:02:30.306600 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260807 23:02:30.307082 30683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:30.292910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35352 (local address 127.25.94.61:38653)
0807 23:02:30.293102 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0807 23:02:30.293108 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 23:02:30.293133 (+ 25us) server_negotiation.cc:407] Connection header received
0807 23:02:30.293309 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:02:30.293315 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:02:30.293415 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:02:30.293541 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:02:30.294327 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:30.295389 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:30.296525 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:30.297273 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:30.297407 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:02:30.300709 (+ 3302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:02:30.300716 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:02:30.300721 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 23:02:30.300762 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:02:30.304344 (+ 3582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:02:30.305027 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:02:30.305034 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:02:30.305040 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:02:30.305228 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:02:30.305584 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:02:30.305588 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:02:30.305592 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:02:30.305984 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:02:30.306127 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0807 23:02:30.306538 (+ 411us) server_negotiation.cc:299] Negotiation successful
0807 23:02:30.306772 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260807 23:02:30.308301 30552 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.94.60" port: 37557 } has no permanent_uuid. Determining permanent_uuid...
I20260807 23:02:30.309326 30557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:30.308891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.60:37557 (local address 127.0.0.1:45916)
0807 23:02:30.309053 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0807 23:02:30.309134 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.60:37557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260807 23:02:30.310310 30557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:30.309989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.60:37557 (local address 127.0.0.1:45924)
0807 23:02:30.310140 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0807 23:02:30.310213 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.60:37557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260807 23:02:30.310600 30552 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.94.60:37557: Network error: Client connection negotiation failed: client connection to 127.25.94.60:37557: connect: Connection refused (error 111)
I20260807 23:02:30.343931 30676 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.94.60" port: 37557 } attempt: 2
I20260807 23:02:30.345265 30680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:30.344648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.60:37557 (local address 127.0.0.1:45932)
0807 23:02:30.344894 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0807 23:02:30.345013 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.60:37557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260807 23:02:30.346391 30680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:30.345988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.60:37557 (local address 127.0.0.1:45938)
0807 23:02:30.346140 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0807 23:02:30.346232 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.60:37557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260807 23:02:30.346698 30676 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.94.60:37557: Network error: Client connection negotiation failed: client connection to 127.25.94.60:37557: connect: Connection refused (error 111)
I20260807 23:02:30.361881 30552 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.94.60" port: 37557 } attempt: 1
I20260807 23:02:30.363157 30557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:30.362651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.60:37557 (local address 127.0.0.1:45950)
0807 23:02:30.362884 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0807 23:02:30.362986 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.60:37557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260807 23:02:30.364355 30557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:30.363972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.60:37557 (local address 127.0.0.1:45952)
0807 23:02:30.364156 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0807 23:02:30.364226 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.60:37557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260807 23:02:30.364588 30552 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.94.60:37557: Network error: Client connection negotiation failed: client connection to 127.25.94.60:37557: connect: Connection refused (error 111)
I20260807 23:02:30.432844 30552 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.94.60" port: 37557 } attempt: 2
I20260807 23:02:30.434158 30557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:30.433630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.60:37557 (local address 127.0.0.1:45966)
0807 23:02:30.433897 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0807 23:02:30.433986 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.60:37557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260807 23:02:30.435441 30557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:30.435048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.60:37557 (local address 127.0.0.1:45978)
0807 23:02:30.435230 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0807 23:02:30.435324 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.60:37557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260807 23:02:30.435726 30552 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.94.60:37557: Network error: Client connection negotiation failed: client connection to 127.25.94.60:37557: connect: Connection refused (error 111)
I20260807 23:02:30.437960 30676 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.94.60" port: 37557 } attempt: 3
I20260807 23:02:30.439152 30680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:30.438581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.60:37557 (local address 127.0.0.1:45994)
0807 23:02:30.438858 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0807 23:02:30.438969 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.60:37557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260807 23:02:30.440316 30680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:30.439834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.60:37557 (local address 127.0.0.1:46008)
0807 23:02:30.440073 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0807 23:02:30.440169 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.60:37557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134}
W20260807 23:02:30.440564 30676 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.94.60:37557: Network error: Client connection negotiation failed: client connection to 127.25.94.60:37557: connect: Connection refused (error 111)
I20260807 23:02:30.541950 30552 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.94.60" port: 37557 } attempt: 3
I20260807 23:02:30.543078 30557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:30.542608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.60:37557 (local address 127.0.0.1:46012)
0807 23:02:30.542805 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0807 23:02:30.542904 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.60:37557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260807 23:02:30.544286 30557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:30.543782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.60:37557 (local address 127.0.0.1:46022)
0807 23:02:30.544022 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0807 23:02:30.544132 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.60:37557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260807 23:02:30.544538 30552 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.94.60:37557: Network error: Client connection negotiation failed: client connection to 127.25.94.60:37557: connect: Connection refused (error 111)
I20260807 23:02:30.583817 30676 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.94.60" port: 37557 } attempt: 4
I20260807 23:02:30.585248 30680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:30.584597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.60:37557 (local address 127.0.0.1:46026)
0807 23:02:30.584890 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0807 23:02:30.585007 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.60:37557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260807 23:02:30.586443 30680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:30.586056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.60:37557 (local address 127.0.0.1:46032)
0807 23:02:30.586204 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0807 23:02:30.586299 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.60:37557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260807 23:02:30.586784 30676 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.94.60:37557: Network error: Client connection negotiation failed: client connection to 127.25.94.60:37557: connect: Connection refused (error 111)
W20260807 23:02:30.609298 30686 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260807 23:02:30.609627 30686 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 23:02:30.609692 30686 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 23:02:30.609773 30686 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 23:02:30.617091 30686 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 23:02:30.617220 30686 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 23:02:30.617285 30686 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 23:02:30.617329 30686 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 23:02:30.617394 30686 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 23:02:30.627808 30686 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:45191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786143361779964-25976-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786143361779964-25976-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.25.94.62:34799,127.25.94.61:38653,127.25.94.60:37557
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.60.keytab
--principal=kudu/127.25.94.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.60:37557
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786143361779964-25976-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.25.94.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786143361779964-25976-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 23:02:30.629119 30686 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 23:02:30.630530 30686 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 23:02:30.642765 30694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 23:02:30.642835 30695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 23:02:30.643432 30700 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 23:02:30.643766 30686 server_base.cc:1034] running on GCE node
I20260807 23:02:30.644667 30686 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 23:02:30.646239 30686 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 23:02:30.647503 30686 hybrid_clock.cc:648] HybridClock initialized: now 1786143750647423 us; error 65 us; skew 500 ppm
I20260807 23:02:30.647877 30686 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 23:02:30 dist-test-slave-z8x9 krb5kdc[30416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143750, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 23:02:30.651099 30686 init.cc:377] Logged in from keytab as kudu/127.25.94.60@KRBTEST.COM (short username kudu)
I20260807 23:02:30.652807 30686 webserver.cc:460] Webserver started at http://127.25.94.60:32799/ using document root <none> and password file <none>
I20260807 23:02:30.653599 30686 fs_manager.cc:359] Metadata directory not provided
I20260807 23:02:30.653748 30686 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 23:02:30.654088 30686 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 23:02:30.657140 30686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786143361779964-25976-0/minicluster-data/master-2/data/instance:
uuid: "3bdd8a084786405ca9faed30265c46be"
format_stamp: "Formatted at 2026-08-07 23:02:30 on dist-test-slave-z8x9"
server_key: "57f4d6db84ec030cfbff5874ae451207"
server_key_iv: "3e6887a71e5ccbe05401c1846ebe6971"
server_key_version: "encryptionkey@0"
I20260807 23:02:30.658221 30686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786143361779964-25976-0/minicluster-data/master-2/wal/instance:
uuid: "3bdd8a084786405ca9faed30265c46be"
format_stamp: "Formatted at 2026-08-07 23:02:30 on dist-test-slave-z8x9"
server_key: "57f4d6db84ec030cfbff5874ae451207"
server_key_iv: "3e6887a71e5ccbe05401c1846ebe6971"
server_key_version: "encryptionkey@0"
I20260807 23:02:30.665519 30686 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.003s
I20260807 23:02:30.669124 30709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 23:02:30.670548 30686 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 23:02:30.670698 30686 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786143361779964-25976-0/minicluster-data/master-2/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786143361779964-25976-0/minicluster-data/master-2/wal
uuid: "3bdd8a084786405ca9faed30265c46be"
format_stamp: "Formatted at 2026-08-07 23:02:30 on dist-test-slave-z8x9"
server_key: "57f4d6db84ec030cfbff5874ae451207"
server_key_iv: "3e6887a71e5ccbe05401c1846ebe6971"
server_key_version: "encryptionkey@0"
I20260807 23:02:30.670914 30686 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786143361779964-25976-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786143361779964-25976-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786143361779964-25976-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 23:02:30.683048 30686 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 23:02:30.687053 30686 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 23:02:30.687284 30686 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 23:02:30.706626 30686 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.60:37557
I20260807 23:02:30.706660 30798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.60:37557 every 8 connection(s)
I20260807 23:02:30.707998 30686 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786143361779964-25976-0/minicluster-data/master-2/data/info.pb
I20260807 23:02:30.708514 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 30686
I20260807 23:02:30.708740 25976 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786143361779964-25976-0/minicluster-data/master-2/wal/instance
I20260807 23:02:30.709425 25976 external_mini_cluster.cc:1428] Setting key 7ddefcf1aec62926d1d5725e846f382d
I20260807 23:02:30.712498 30800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 23:02:30.721836 30552 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.94.60" port: 37557 } attempt: 4
I20260807 23:02:30.720999 30800 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.94.62" port: 34799 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 23:02:30 dist-test-slave-z8x9 krb5kdc[30416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143748, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.60@KRBTEST.COM
Aug 07 23:02:30 dist-test-slave-z8x9 krb5kdc[30416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143749, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.62@KRBTEST.COM for kudu/127.25.94.60@KRBTEST.COM
Aug 07 23:02:30 dist-test-slave-z8x9 krb5kdc[30416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143750, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.60@KRBTEST.COM for kudu/127.25.94.62@KRBTEST.COM
I20260807 23:02:30.737349 30806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:30.711283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46038 (local address 127.25.94.60:37557)
0807 23:02:30.712055 (+ 772us) server_negotiation.cc:206] Beginning negotiation
0807 23:02:30.712067 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 23:02:30.712107 (+ 40us) server_negotiation.cc:407] Connection header received
0807 23:02:30.713876 (+ 1769us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:02:30.713897 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:02:30.714234 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:02:30.714543 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:02:30.716405 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:30.717886 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:30.719716 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:30.720428 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:30.720563 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:02:30.726398 (+ 5835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:02:30.726413 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:02:30.726433 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 23:02:30.726485 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:02:30.730938 (+ 4453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:02:30.731920 (+ 982us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:02:30.731928 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:02:30.731937 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:02:30.732172 (+ 235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:02:30.732680 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:02:30.732685 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:02:30.732689 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:02:30.733493 (+ 804us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:02:30.735035 (+ 1542us) server_negotiation.cc:1035] Waiting for connection context
0807 23:02:30.735319 (+ 284us) server_negotiation.cc:299] Negotiation successful
0807 23:02:30.735741 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":571,"thread_start_us":379,"threads_started":1}
I20260807 23:02:30.740016 30557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:30.722632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.60:37557 (local address 127.0.0.1:46050)
0807 23:02:30.722939 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0807 23:02:30.722963 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0807 23:02:30.723093 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 23:02:30.724082 (+ 989us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 23:02:30.724089 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 23:02:30.724677 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0807 23:02:30.724909 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:02:30.724923 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:02:30.727071 (+ 2148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:02:30.727077 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:02:30.728257 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:02:30.728269 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:02:30.729297 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:02:30.729303 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:02:30.729560 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:02:30.730204 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 23:02:30.730229 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0807 23:02:30.733160 (+ 2931us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 23:02:30.737218 (+ 4058us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 23:02:30.737224 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 23:02:30.737230 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 23:02:30.737787 (+ 557us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 23:02:30.738151 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 23:02:30.738156 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 23:02:30.738160 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 23:02:30.738301 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 23:02:30.738930 (+ 629us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 23:02:30.738935 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 23:02:30.739126 (+ 191us) client_negotiation.cc:769] Sending connection context
0807 23:02:30.739331 (+ 205us) client_negotiation.cc:240] Negotiation successful
0807 23:02:30.739643 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178}
I20260807 23:02:30.740052 30809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:30.722785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46050 (local address 127.25.94.60:37557)
0807 23:02:30.723721 (+ 936us) server_negotiation.cc:206] Beginning negotiation
0807 23:02:30.723729 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 23:02:30.723756 (+ 27us) server_negotiation.cc:407] Connection header received
0807 23:02:30.723808 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:02:30.723819 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:02:30.723890 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:02:30.724009 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:02:30.725619 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:30.726905 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:30.728397 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:30.729106 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:30.729259 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:02:30.733424 (+ 4165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:02:30.733432 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:02:30.733438 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 23:02:30.733487 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:02:30.737045 (+ 3558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:02:30.737891 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:02:30.737898 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:02:30.737901 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:02:30.738035 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:02:30.738417 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:02:30.738423 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:02:30.738426 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:02:30.738754 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:02:30.738978 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0807 23:02:30.739414 (+ 436us) server_negotiation.cc:299] Negotiation successful
0807 23:02:30.739620 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":767,"thread_start_us":603,"threads_started":1}
I20260807 23:02:30.744923 30812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:30.724363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:34799 (local address 127.0.0.1:32962)
0807 23:02:30.725219 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0807 23:02:30.725273 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0807 23:02:30.725408 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 23:02:30.726881 (+ 1473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 23:02:30.726889 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 23:02:30.727920 (+ 1031us) client_negotiation.cc:189] Negotiated authn=SASL
0807 23:02:30.728311 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:02:30.728327 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:02:30.729683 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:02:30.729688 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:02:30.730924 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:02:30.730936 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:02:30.731743 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:02:30.731750 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:02:30.731939 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:02:30.732584 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 23:02:30.732607 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0807 23:02:30.735415 (+ 2808us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 23:02:30.741627 (+ 6212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 23:02:30.741639 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 23:02:30.741646 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0807 23:02:30.742196 (+ 550us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 23:02:30.742647 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 23:02:30.742653 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 23:02:30.742657 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 23:02:30.742902 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 23:02:30.743749 (+ 847us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 23:02:30.743758 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 23:02:30.743943 (+ 185us) client_negotiation.cc:769] Sending connection context
0807 23:02:30.744147 (+ 204us) client_negotiation.cc:240] Negotiation successful
0807 23:02:30.744499 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":427,"threads_started":1}
I20260807 23:02:30.745034 30813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:30.724550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32962 (local address 127.25.94.62:34799)
0807 23:02:30.726547 (+ 1997us) server_negotiation.cc:206] Beginning negotiation
0807 23:02:30.726553 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 23:02:30.726569 (+ 16us) server_negotiation.cc:407] Connection header received
0807 23:02:30.726615 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:02:30.726620 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:02:30.726693 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:02:30.726792 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:02:30.728456 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:30.729497 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:30.731096 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:30.731554 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:30.731693 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:02:30.735638 (+ 3945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:02:30.735648 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:02:30.735654 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 23:02:30.735706 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:02:30.741433 (+ 5727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:02:30.742355 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:02:30.742363 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:02:30.742370 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:02:30.742510 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:02:30.743040 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:02:30.743046 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:02:30.743049 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:02:30.743491 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:02:30.743660 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0807 23:02:30.744308 (+ 648us) server_negotiation.cc:299] Negotiation successful
0807 23:02:30.744524 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1883,"thread_start_us":1766,"threads_started":1}
I20260807 23:02:30.746795 30552 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a6b04948f9c45ce99e9a420c6b2563b: Bootstrap starting.
I20260807 23:02:30.747701 30800 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.94.61" port: 38653 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 23:02:30 dist-test-slave-z8x9 krb5kdc[30416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143748, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.62@KRBTEST.COM
I20260807 23:02:30.752007 30552 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1a6b04948f9c45ce99e9a420c6b2563b: Neither blocks nor log segments found. Creating new log.
I20260807 23:02:30.753404 30552 log.cc:824] T 00000000000000000000000000000000 P 1a6b04948f9c45ce99e9a420c6b2563b: Log is configured to *not* fsync() on all Append() calls
Aug 07 23:02:30 dist-test-slave-z8x9 krb5kdc[30416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143750, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.60@KRBTEST.COM for kudu/127.25.94.61@KRBTEST.COM
I20260807 23:02:30.756951 30552 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a6b04948f9c45ce99e9a420c6b2563b: No bootstrap required, opened a new log
I20260807 23:02:30.760655 30816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:30.741075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32968 (local address 127.25.94.62:34799)
0807 23:02:30.741763 (+ 688us) server_negotiation.cc:206] Beginning negotiation
0807 23:02:30.741768 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 23:02:30.741785 (+ 17us) server_negotiation.cc:407] Connection header received
0807 23:02:30.741822 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:02:30.741825 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:02:30.741877 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:02:30.741999 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:02:30.743562 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:30.744764 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:30.746608 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:30.747313 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:30.747441 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:02:30.752977 (+ 5536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:02:30.752982 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:02:30.752985 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 23:02:30.753017 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:02:30.757311 (+ 4294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:02:30.758223 (+ 912us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:02:30.758232 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:02:30.758238 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:02:30.758431 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:02:30.758865 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:02:30.758870 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:02:30.758874 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:02:30.759211 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:02:30.759543 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0807 23:02:30.759872 (+ 329us) server_negotiation.cc:299] Negotiation successful
0807 23:02:30.760192 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":535,"thread_start_us":301,"threads_started":1}
I20260807 23:02:30.764487 30683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:30.748613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35360 (local address 127.25.94.61:38653)
0807 23:02:30.748833 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0807 23:02:30.748839 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 23:02:30.748944 (+ 105us) server_negotiation.cc:407] Connection header received
0807 23:02:30.749125 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:02:30.749131 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:02:30.749261 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:02:30.749378 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:02:30.750087 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:30.750791 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:30.752579 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:30.753287 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:30.753439 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:02:30.757577 (+ 4138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:02:30.757590 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:02:30.757593 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 23:02:30.757627 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:02:30.761212 (+ 3585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:02:30.762043 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:02:30.762048 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:02:30.762052 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:02:30.762211 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:02:30.762617 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:02:30.762623 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:02:30.762627 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:02:30.763015 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:02:30.763184 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0807 23:02:30.763830 (+ 646us) server_negotiation.cc:299] Negotiation successful
0807 23:02:30.764064 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260807 23:02:30.764667 30812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:30.748609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.61:38653 (local address 127.0.0.1:35360)
0807 23:02:30.748851 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0807 23:02:30.748866 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 23:02:30.748996 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 23:02:30.749402 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 23:02:30.749406 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 23:02:30.749808 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0807 23:02:30.749944 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:02:30.749952 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:02:30.750946 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:02:30.750951 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:02:30.752408 (+ 1457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:02:30.752424 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:02:30.753460 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:02:30.753468 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:02:30.753662 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:02:30.754449 (+ 787us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 23:02:30.754472 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0807 23:02:30.757343 (+ 2871us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 23:02:30.761403 (+ 4060us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 23:02:30.761409 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 23:02:30.761413 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 23:02:30.761920 (+ 507us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 23:02:30.762341 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 23:02:30.762346 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 23:02:30.762349 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 23:02:30.762502 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 23:02:30.763283 (+ 781us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 23:02:30.763291 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 23:02:30.763518 (+ 227us) client_negotiation.cc:769] Sending connection context
0807 23:02:30.763710 (+ 192us) client_negotiation.cc:240] Negotiation successful
0807 23:02:30.764120 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148}
I20260807 23:02:30.768805 30552 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1a6b04948f9c45ce99e9a420c6b2563b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a6b04948f9c45ce99e9a420c6b2563b" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 34799 } } peers { permanent_uuid: "9bcfdd00153040c7ae21f38a2ffaf010" member_type: VOTER last_known_addr { host: "127.25.94.61" port: 38653 } } peers { permanent_uuid: "3bdd8a084786405ca9faed30265c46be" member_type: VOTER last_known_addr { host: "127.25.94.60" port: 37557 } }
I20260807 23:02:30.769279 30552 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1a6b04948f9c45ce99e9a420c6b2563b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a6b04948f9c45ce99e9a420c6b2563b, State: Initialized, Role: FOLLOWER
I20260807 23:02:30.769842 30552 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a6b04948f9c45ce99e9a420c6b2563b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a6b04948f9c45ce99e9a420c6b2563b" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 34799 } } peers { permanent_uuid: "9bcfdd00153040c7ae21f38a2ffaf010" member_type: VOTER last_known_addr { host: "127.25.94.61" port: 38653 } } peers { permanent_uuid: "3bdd8a084786405ca9faed30265c46be" member_type: VOTER last_known_addr { host: "127.25.94.60" port: 37557 } }
I20260807 23:02:30.770701 30800 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.94.60" port: 37557 } has no permanent_uuid. Determining permanent_uuid...
I20260807 23:02:30.771821 30819 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a6b04948f9c45ce99e9a420c6b2563b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a6b04948f9c45ce99e9a420c6b2563b" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 34799 } } peers { permanent_uuid: "9bcfdd00153040c7ae21f38a2ffaf010" member_type: VOTER last_known_addr { host: "127.25.94.61" port: 38653 } } peers { permanent_uuid: "3bdd8a084786405ca9faed30265c46be" member_type: VOTER last_known_addr { host: "127.25.94.60" port: 37557 } } }
I20260807 23:02:30.772383 30819 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a6b04948f9c45ce99e9a420c6b2563b [sys.catalog]: This master's current role is: FOLLOWER
I20260807 23:02:30.773496 30552 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1a6b04948f9c45ce99e9a420c6b2563b [sys.catalog]: configured and running, proceeding with master startup.
Aug 07 23:02:30 dist-test-slave-z8x9 krb5kdc[30416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143748, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.61@KRBTEST.COM
Aug 07 23:02:30 dist-test-slave-z8x9 krb5kdc[30416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143750, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.60@KRBTEST.COM for kudu/127.25.94.60@KRBTEST.COM
I20260807 23:02:30.785363 30683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:30.762933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35376 (local address 127.25.94.61:38653)
0807 23:02:30.765098 (+ 2165us) server_negotiation.cc:206] Beginning negotiation
0807 23:02:30.765104 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 23:02:30.765123 (+ 19us) server_negotiation.cc:407] Connection header received
0807 23:02:30.765208 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:02:30.765213 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:02:30.765288 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:02:30.765381 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:02:30.767155 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:30.768350 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:30.770160 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:30.770751 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:30.770888 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:02:30.777399 (+ 6511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:02:30.777404 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:02:30.777408 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 23:02:30.777442 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:02:30.781333 (+ 3891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:02:30.782391 (+ 1058us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:02:30.782397 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:02:30.782401 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:02:30.782590 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:02:30.783125 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:02:30.783130 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:02:30.783133 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:02:30.783519 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:02:30.783829 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0807 23:02:30.784664 (+ 835us) server_negotiation.cc:299] Negotiation successful
0807 23:02:30.784891 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2064,"thread_start_us":6676,"threads_started":1}
W20260807 23:02:30.788647 30838 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1a6b04948f9c45ce99e9a420c6b2563b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 23:02:30.788782 30838 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 23:02:30.789621 30812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:30.771495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.60:37557 (local address 127.0.0.1:46062)
0807 23:02:30.771868 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0807 23:02:30.771895 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0807 23:02:30.772007 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 23:02:30.772771 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 23:02:30.772774 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 23:02:30.773135 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0807 23:02:30.773363 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:02:30.773372 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:02:30.774559 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:02:30.774567 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:02:30.776034 (+ 1467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:02:30.776053 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:02:30.776777 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:02:30.776783 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:02:30.777023 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:02:30.777787 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 23:02:30.777811 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0807 23:02:30.780521 (+ 2710us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 23:02:30.786007 (+ 5486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 23:02:30.786012 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 23:02:30.786015 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 23:02:30.786564 (+ 549us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 23:02:30.787005 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 23:02:30.787009 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 23:02:30.787013 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 23:02:30.787184 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 23:02:30.787918 (+ 734us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 23:02:30.787921 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 23:02:30.788085 (+ 164us) client_negotiation.cc:769] Sending connection context
0807 23:02:30.788260 (+ 175us) client_negotiation.cc:240] Negotiation successful
0807 23:02:30.788583 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"mutex_wait_us":1}
I20260807 23:02:30.789698 30809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:30.772201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46062 (local address 127.25.94.60:37557)
0807 23:02:30.772356 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0807 23:02:30.772363 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 23:02:30.772383 (+ 20us) server_negotiation.cc:407] Connection header received
0807 23:02:30.772431 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:02:30.772436 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:02:30.772529 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:02:30.772627 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:02:30.773497 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:30.774436 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:30.776195 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:30.776614 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:30.776761 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:02:30.780744 (+ 3983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:02:30.780751 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:02:30.780756 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 23:02:30.780796 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:02:30.784402 (+ 3606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:02:30.786686 (+ 2284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:02:30.786691 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:02:30.786694 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:02:30.786859 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:02:30.787295 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:02:30.787301 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:02:30.787303 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:02:30.787697 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:02:30.787850 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0807 23:02:30.788985 (+ 1135us) server_negotiation.cc:299] Negotiation successful
0807 23:02:30.789229 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":295,"server-negotiator.queue_time_us":61}
I20260807 23:02:30.795280 30800 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bdd8a084786405ca9faed30265c46be: Bootstrap starting.
I20260807 23:02:30.796180 25976 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 23:02:30.786579 30436 master_runner.cc:186] Error getting master registration for 127.25.94.61:38653: OK, instance_id { permanent_uuid: "9bcfdd00153040c7ae21f38a2ffaf010" instance_seqno: 1786143750140510 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260807 23:02:30.797448 30436 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260807 23:02:30.798795 30800 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3bdd8a084786405ca9faed30265c46be: Neither blocks nor log segments found. Creating new log.
I20260807 23:02:30.799952 30800 log.cc:824] T 00000000000000000000000000000000 P 3bdd8a084786405ca9faed30265c46be: Log is configured to *not* fsync() on all Append() calls
I20260807 23:02:30.800045 30515 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "1a6b04948f9c45ce99e9a420c6b2563b"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32968
I20260807 23:02:30.800565 30515 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1a6b04948f9c45ce99e9a420c6b2563b [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 23:02:30.800792 30515 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1a6b04948f9c45ce99e9a420c6b2563b [term 0 FOLLOWER]: Advancing to term 1
I20260807 23:02:30.802834 30800 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bdd8a084786405ca9faed30265c46be: No bootstrap required, opened a new log
I20260807 23:02:30.804344 30515 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1a6b04948f9c45ce99e9a420c6b2563b [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a6b04948f9c45ce99e9a420c6b2563b" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 34799 } } peers { permanent_uuid: "9bcfdd00153040c7ae21f38a2ffaf010" member_type: VOTER last_known_addr { host: "127.25.94.61" port: 38653 } } peers { permanent_uuid: "3bdd8a084786405ca9faed30265c46be" member_type: VOTER last_known_addr { host: "127.25.94.60" port: 37557 } }
I20260807 23:02:30.805971 30515 leader_election.cc:288] T 00000000000000000000000000000000 P 1a6b04948f9c45ce99e9a420c6b2563b [CANDIDATE]: Term 1 election: Requested vote from peers 9bcfdd00153040c7ae21f38a2ffaf010 (127.25.94.61:38653), 3bdd8a084786405ca9faed30265c46be (127.25.94.60:37557)
W20260807 23:02:30.812914 30480 tablet.cc:2369] T 00000000000000000000000000000000 P 1a6b04948f9c45ce99e9a420c6b2563b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 23:02:30.812960 30800 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3bdd8a084786405ca9faed30265c46be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a6b04948f9c45ce99e9a420c6b2563b" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 34799 } } peers { permanent_uuid: "9bcfdd00153040c7ae21f38a2ffaf010" member_type: VOTER last_known_addr { host: "127.25.94.61" port: 38653 } } peers { permanent_uuid: "3bdd8a084786405ca9faed30265c46be" member_type: VOTER last_known_addr { host: "127.25.94.60" port: 37557 } }
I20260807 23:02:30.813508 30800 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3bdd8a084786405ca9faed30265c46be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bdd8a084786405ca9faed30265c46be, State: Initialized, Role: FOLLOWER
I20260807 23:02:30.814080 30800 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3bdd8a084786405ca9faed30265c46be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a6b04948f9c45ce99e9a420c6b2563b" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 34799 } } peers { permanent_uuid: "9bcfdd00153040c7ae21f38a2ffaf010" member_type: VOTER last_known_addr { host: "127.25.94.61" port: 38653 } } peers { permanent_uuid: "3bdd8a084786405ca9faed30265c46be" member_type: VOTER last_known_addr { host: "127.25.94.60" port: 37557 } }
I20260807 23:02:30.816555 30800 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3bdd8a084786405ca9faed30265c46be [sys.catalog]: configured and running, proceeding with master startup.
I20260807 23:02:30.816732 30761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1a6b04948f9c45ce99e9a420c6b2563b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "3bdd8a084786405ca9faed30265c46be"
I20260807 23:02:30.824502 30761 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3bdd8a084786405ca9faed30265c46be [term 0 FOLLOWER]: Advancing to term 1
I20260807 23:02:30.830235 30761 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3bdd8a084786405ca9faed30265c46be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a6b04948f9c45ce99e9a420c6b2563b in term 1.
I20260807 23:02:30.833490 30474 leader_election.cc:302] T 00000000000000000000000000000000 P 1a6b04948f9c45ce99e9a420c6b2563b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a6b04948f9c45ce99e9a420c6b2563b, 3bdd8a084786405ca9faed30265c46be; no voters:
I20260807 23:02:30.834153 30819 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1a6b04948f9c45ce99e9a420c6b2563b [term 1 FOLLOWER]: Leader election won for term 1
I20260807 23:02:30.834861 30819 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1a6b04948f9c45ce99e9a420c6b2563b [term 1 LEADER]: Becoming Leader. State: Replica: 1a6b04948f9c45ce99e9a420c6b2563b, State: Running, Role: LEADER
I20260807 23:02:30.835511 30819 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a6b04948f9c45ce99e9a420c6b2563b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a6b04948f9c45ce99e9a420c6b2563b" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 34799 } } peers { permanent_uuid: "9bcfdd00153040c7ae21f38a2ffaf010" member_type: VOTER last_known_addr { host: "127.25.94.61" port: 38653 } } peers { permanent_uuid: "3bdd8a084786405ca9faed30265c46be" member_type: VOTER last_known_addr { host: "127.25.94.60" port: 37557 } }
I20260807 23:02:30.839032 30848 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bdd8a084786405ca9faed30265c46be [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a6b04948f9c45ce99e9a420c6b2563b" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 34799 } } peers { permanent_uuid: "9bcfdd00153040c7ae21f38a2ffaf010" member_type: VOTER last_known_addr { host: "127.25.94.61" port: 38653 } } peers { permanent_uuid: "3bdd8a084786405ca9faed30265c46be" member_type: VOTER last_known_addr { host: "127.25.94.60" port: 37557 } } }
I20260807 23:02:30.841068 30848 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bdd8a084786405ca9faed30265c46be [sys.catalog]: This master's current role is: FOLLOWER
I20260807 23:02:30.841747 30860 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a6b04948f9c45ce99e9a420c6b2563b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a6b04948f9c45ce99e9a420c6b2563b. Latest consensus state: current_term: 1 leader_uuid: "1a6b04948f9c45ce99e9a420c6b2563b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a6b04948f9c45ce99e9a420c6b2563b" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 34799 } } peers { permanent_uuid: "9bcfdd00153040c7ae21f38a2ffaf010" member_type: VOTER last_known_addr { host: "127.25.94.61" port: 38653 } } peers { permanent_uuid: "3bdd8a084786405ca9faed30265c46be" member_type: VOTER last_known_addr { host: "127.25.94.60" port: 37557 } } }
I20260807 23:02:30.842298 30860 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a6b04948f9c45ce99e9a420c6b2563b [sys.catalog]: This master's current role is: LEADER
I20260807 23:02:30.848407 30863 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 23:02:30.850971 30863 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 23:02:30.873966 30761 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3bdd8a084786405ca9faed30265c46be [term 1 FOLLOWER]: Refusing update from remote peer 1a6b04948f9c45ce99e9a420c6b2563b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 23:02:30.875226 30860 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1a6b04948f9c45ce99e9a420c6b2563b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bdd8a084786405ca9faed30265c46be" member_type: VOTER last_known_addr { host: "127.25.94.60" port: 37557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 23:02:30.886027 30676 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.94.60" port: 37557 } attempt: 5
I20260807 23:02:30.890156 30816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:30.816369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32980 (local address 127.25.94.62:34799)
0807 23:02:30.816565 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0807 23:02:30.816574 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 23:02:30.846066 (+ 29492us) server_negotiation.cc:407] Connection header received
0807 23:02:30.846589 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:02:30.846598 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:02:30.846710 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:02:30.846798 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:02:30.854112 (+ 7314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:30.855269 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:30.862077 (+ 6808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:30.863727 (+ 1650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:30.863864 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:02:30.874271 (+ 10407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:02:30.874278 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:02:30.874283 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 23:02:30.874327 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:02:30.878320 (+ 3993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:02:30.881561 (+ 3241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:02:30.881566 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:02:30.881569 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:02:30.881741 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:02:30.883806 (+ 2065us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:02:30.883815 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:02:30.883821 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:02:30.884360 (+ 539us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:02:30.884668 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0807 23:02:30.889511 (+ 4843us) server_negotiation.cc:299] Negotiation successful
0807 23:02:30.889787 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260807 23:02:30.896878 30848 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bdd8a084786405ca9faed30265c46be [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a6b04948f9c45ce99e9a420c6b2563b. Latest consensus state: current_term: 1 leader_uuid: "1a6b04948f9c45ce99e9a420c6b2563b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a6b04948f9c45ce99e9a420c6b2563b" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 34799 } } peers { permanent_uuid: "9bcfdd00153040c7ae21f38a2ffaf010" member_type: VOTER last_known_addr { host: "127.25.94.61" port: 38653 } } peers { permanent_uuid: "3bdd8a084786405ca9faed30265c46be" member_type: VOTER last_known_addr { host: "127.25.94.60" port: 37557 } } }
I20260807 23:02:30.897292 30848 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bdd8a084786405ca9faed30265c46be [sys.catalog]: This master's current role is: FOLLOWER
Aug 07 23:02:30 dist-test-slave-z8x9 krb5kdc[30416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143750, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.61@KRBTEST.COM for kudu/127.25.94.60@KRBTEST.COM
I20260807 23:02:30.899557 30867 mvcc.cc:205] Tried to move back new op lower bound from 7316044803563958272 to 7316044803428421632. Current Snapshot: MvccSnapshot[applied={T|T < 7316044803563958272}]
I20260807 23:02:30.900279 30872 mvcc.cc:205] Tried to move back new op lower bound from 7316044803563958272 to 7316044803428421632. Current Snapshot: MvccSnapshot[applied={T|T < 7316044803563958272}]
I20260807 23:02:30.910822 30819 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a6b04948f9c45ce99e9a420c6b2563b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1a6b04948f9c45ce99e9a420c6b2563b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a6b04948f9c45ce99e9a420c6b2563b" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 34799 } } peers { permanent_uuid: "9bcfdd00153040c7ae21f38a2ffaf010" member_type: VOTER last_known_addr { host: "127.25.94.61" port: 38653 } } peers { permanent_uuid: "3bdd8a084786405ca9faed30265c46be" member_type: VOTER last_known_addr { host: "127.25.94.60" port: 37557 } } }
I20260807 23:02:30.911194 30819 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a6b04948f9c45ce99e9a420c6b2563b [sys.catalog]: This master's current role is: LEADER
I20260807 23:02:30.911386 30680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:30.886842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.60:37557 (local address 127.0.0.1:46086)
0807 23:02:30.887282 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0807 23:02:30.887302 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 23:02:30.887415 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 23:02:30.888447 (+ 1032us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 23:02:30.888455 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 23:02:30.889134 (+ 679us) client_negotiation.cc:189] Negotiated authn=SASL
0807 23:02:30.889423 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:02:30.889441 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:02:30.891262 (+ 1821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:02:30.891270 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:02:30.893251 (+ 1981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:02:30.893273 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:02:30.894888 (+ 1615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:02:30.894897 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:02:30.895191 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:02:30.895991 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 23:02:30.896023 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0807 23:02:30.899942 (+ 3919us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 23:02:30.905421 (+ 5479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 23:02:30.905429 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 23:02:30.905435 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 23:02:30.905989 (+ 554us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 23:02:30.906798 (+ 809us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 23:02:30.906831 (+ 33us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 23:02:30.906834 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 23:02:30.907189 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 23:02:30.908578 (+ 1389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 23:02:30.908614 (+ 36us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 23:02:30.908893 (+ 279us) client_negotiation.cc:769] Sending connection context
0807 23:02:30.909153 (+ 260us) client_negotiation.cc:240] Negotiation successful
0807 23:02:30.909694 (+ 541us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317}
I20260807 23:02:30.912408 30683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:30.816483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35392 (local address 127.25.94.61:38653)
0807 23:02:30.816676 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0807 23:02:30.816683 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 23:02:30.851337 (+ 34654us) server_negotiation.cc:407] Connection header received
0807 23:02:30.852156 (+ 819us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:02:30.852166 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:02:30.852279 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:02:30.852371 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:02:30.860809 (+ 8438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:30.862112 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:30.867589 (+ 5477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:30.868268 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:30.868384 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:02:30.891670 (+ 23286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:02:30.891681 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:02:30.891689 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 23:02:30.891765 (+ 76us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:02:30.898968 (+ 7203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:02:30.907722 (+ 8754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:02:30.907732 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:02:30.907738 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:02:30.907895 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:02:30.909993 (+ 2098us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:02:30.909999 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:02:30.910002 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:02:30.910378 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:02:30.910634 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0807 23:02:30.911661 (+ 1027us) server_negotiation.cc:299] Negotiation successful
0807 23:02:30.912071 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260807 23:02:30.912914 30806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:30.887631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46086 (local address 127.25.94.60:37557)
0807 23:02:30.887855 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0807 23:02:30.887862 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 23:02:30.887889 (+ 27us) server_negotiation.cc:407] Connection header received
0807 23:02:30.887956 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:02:30.887964 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:02:30.888063 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:02:30.888193 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:02:30.889642 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:30.891080 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:30.893470 (+ 2390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:30.894467 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:30.894629 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:02:30.900181 (+ 5552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:02:30.900191 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:02:30.900199 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 23:02:30.900251 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:02:30.905185 (+ 4934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:02:30.906181 (+ 996us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:02:30.906187 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:02:30.906220 (+ 33us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:02:30.906559 (+ 339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:02:30.907468 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:02:30.907475 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:02:30.907478 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:02:30.908340 (+ 862us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:02:30.910503 (+ 2163us) server_negotiation.cc:1035] Waiting for connection context
0807 23:02:30.911048 (+ 545us) server_negotiation.cc:299] Negotiation successful
0807 23:02:30.912104 (+ 1056us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260807 23:02:30.915355 30848 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bdd8a084786405ca9faed30265c46be [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1a6b04948f9c45ce99e9a420c6b2563b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a6b04948f9c45ce99e9a420c6b2563b" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 34799 } } peers { permanent_uuid: "9bcfdd00153040c7ae21f38a2ffaf010" member_type: VOTER last_known_addr { host: "127.25.94.61" port: 38653 } } peers { permanent_uuid: "3bdd8a084786405ca9faed30265c46be" member_type: VOTER last_known_addr { host: "127.25.94.60" port: 37557 } } }
I20260807 23:02:30.915674 30848 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bdd8a084786405ca9faed30265c46be [sys.catalog]: This master's current role is: FOLLOWER
I20260807 23:02:30.919744 30676 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9bcfdd00153040c7ae21f38a2ffaf010: Bootstrap starting.
I20260807 23:02:30.920131 30863 catalog_manager.cc:1347] Generated new cluster ID: 8a07604fcfc94cdf9155a2e2b0a15a7a
I20260807 23:02:30.920266 30863 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 23:02:30.923892 30676 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9bcfdd00153040c7ae21f38a2ffaf010: Neither blocks nor log segments found. Creating new log.
I20260807 23:02:30.925235 30676 log.cc:824] T 00000000000000000000000000000000 P 9bcfdd00153040c7ae21f38a2ffaf010: Log is configured to *not* fsync() on all Append() calls
I20260807 23:02:30.929180 30809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:30.850619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46072 (local address 127.25.94.60:37557)
0807 23:02:30.850822 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0807 23:02:30.850828 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 23:02:30.862196 (+ 11368us) server_negotiation.cc:407] Connection header received
0807 23:02:30.862373 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:02:30.862381 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:02:30.862487 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:02:30.862596 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:02:30.865611 (+ 3015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:30.866680 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:30.879358 (+ 12678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:30.880173 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:30.880296 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:02:30.913400 (+ 33104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:02:30.913406 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:02:30.913411 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 23:02:30.913465 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:02:30.917510 (+ 4045us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:02:30.925391 (+ 7881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:02:30.925403 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:02:30.925409 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:02:30.925782 (+ 373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:02:30.926620 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:02:30.926626 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:02:30.926632 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:02:30.927146 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:02:30.927506 (+ 360us) server_negotiation.cc:1035] Waiting for connection context
0807 23:02:30.928275 (+ 769us) server_negotiation.cc:299] Negotiation successful
0807 23:02:30.928626 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260807 23:02:30.935237 30676 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9bcfdd00153040c7ae21f38a2ffaf010: No bootstrap required, opened a new log
I20260807 23:02:30.952067 30863 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 23:02:30.954082 30863 catalog_manager.cc:1504] Loading token signing keys...
I20260807 23:02:30.956640 30882 catalog_manager.cc:1259] Loaded cluster ID: 8a07604fcfc94cdf9155a2e2b0a15a7a
I20260807 23:02:30.956749 30882 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3bdd8a084786405ca9faed30265c46be: loading cluster ID for follower catalog manager: success
I20260807 23:02:30.960764 30882 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3bdd8a084786405ca9faed30265c46be: acquiring CA information for follower catalog manager: success
W20260807 23:02:30.962167 30882 catalog_manager.cc:873] Not found: no TSK found in the system table: failed to prepare follower catalog manager, will retry
I20260807 23:02:30.945037 30686 master_runner.cc:186] Error getting master registration for 127.25.94.61:38653: OK, instance_id { permanent_uuid: "9bcfdd00153040c7ae21f38a2ffaf010" instance_seqno: 1786143750140510 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260807 23:02:30.967238 30686 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260807 23:02:30.970477 30863 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1a6b04948f9c45ce99e9a420c6b2563b: Generated new TSK 0
I20260807 23:02:30.971619 30863 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 23:02:30.977044 30676 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9bcfdd00153040c7ae21f38a2ffaf010 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a6b04948f9c45ce99e9a420c6b2563b" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 34799 } } peers { permanent_uuid: "9bcfdd00153040c7ae21f38a2ffaf010" member_type: VOTER last_known_addr { host: "127.25.94.61" port: 38653 } } peers { permanent_uuid: "3bdd8a084786405ca9faed30265c46be" member_type: VOTER last_known_addr { host: "127.25.94.60" port: 37557 } }
I20260807 23:02:30.977912 30676 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9bcfdd00153040c7ae21f38a2ffaf010 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bcfdd00153040c7ae21f38a2ffaf010, State: Initialized, Role: FOLLOWER
I20260807 23:02:30.978417 30676 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9bcfdd00153040c7ae21f38a2ffaf010 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a6b04948f9c45ce99e9a420c6b2563b" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 34799 } } peers { permanent_uuid: "9bcfdd00153040c7ae21f38a2ffaf010" member_type: VOTER last_known_addr { host: "127.25.94.61" port: 38653 } } peers { permanent_uuid: "3bdd8a084786405ca9faed30265c46be" member_type: VOTER last_known_addr { host: "127.25.94.60" port: 37557 } }
I20260807 23:02:30.979511 30887 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9bcfdd00153040c7ae21f38a2ffaf010 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a6b04948f9c45ce99e9a420c6b2563b" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 34799 } } peers { permanent_uuid: "9bcfdd00153040c7ae21f38a2ffaf010" member_type: VOTER last_known_addr { host: "127.25.94.61" port: 38653 } } peers { permanent_uuid: "3bdd8a084786405ca9faed30265c46be" member_type: VOTER last_known_addr { host: "127.25.94.60" port: 37557 } } }
I20260807 23:02:30.980036 30887 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9bcfdd00153040c7ae21f38a2ffaf010 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 23:02:30.983232 30676 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9bcfdd00153040c7ae21f38a2ffaf010 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 23:02:30.983319 30637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1a6b04948f9c45ce99e9a420c6b2563b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "9bcfdd00153040c7ae21f38a2ffaf010"
I20260807 23:02:30.983814 30637 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9bcfdd00153040c7ae21f38a2ffaf010 [term 0 FOLLOWER]: Advancing to term 1
I20260807 23:02:30.987365 30637 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9bcfdd00153040c7ae21f38a2ffaf010 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a6b04948f9c45ce99e9a420c6b2563b in term 1.
I20260807 23:02:30.988297 30636 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9bcfdd00153040c7ae21f38a2ffaf010 [term 1 FOLLOWER]: Refusing update from remote peer 1a6b04948f9c45ce99e9a420c6b2563b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 23:02:30.989624 30860 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1a6b04948f9c45ce99e9a420c6b2563b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bcfdd00153040c7ae21f38a2ffaf010" member_type: VOTER last_known_addr { host: "127.25.94.61" port: 38653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 23:02:31.055876 30900 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9bcfdd00153040c7ae21f38a2ffaf010 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1a6b04948f9c45ce99e9a420c6b2563b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a6b04948f9c45ce99e9a420c6b2563b" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 34799 } } peers { permanent_uuid: "9bcfdd00153040c7ae21f38a2ffaf010" member_type: VOTER last_known_addr { host: "127.25.94.61" port: 38653 } } peers { permanent_uuid: "3bdd8a084786405ca9faed30265c46be" member_type: VOTER last_known_addr { host: "127.25.94.60" port: 37557 } } }
I20260807 23:02:31.056303 30900 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9bcfdd00153040c7ae21f38a2ffaf010 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 23:02:31.056636 30887 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9bcfdd00153040c7ae21f38a2ffaf010 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a6b04948f9c45ce99e9a420c6b2563b. Latest consensus state: current_term: 1 leader_uuid: "1a6b04948f9c45ce99e9a420c6b2563b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a6b04948f9c45ce99e9a420c6b2563b" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 34799 } } peers { permanent_uuid: "9bcfdd00153040c7ae21f38a2ffaf010" member_type: VOTER last_known_addr { host: "127.25.94.61" port: 38653 } } peers { permanent_uuid: "3bdd8a084786405ca9faed30265c46be" member_type: VOTER last_known_addr { host: "127.25.94.60" port: 37557 } } }
I20260807 23:02:31.056819 30887 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9bcfdd00153040c7ae21f38a2ffaf010 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 23:02:31.088816 30914 catalog_manager.cc:1259] Loaded cluster ID: 8a07604fcfc94cdf9155a2e2b0a15a7a
I20260807 23:02:31.088971 30914 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9bcfdd00153040c7ae21f38a2ffaf010: loading cluster ID for follower catalog manager: success
W20260807 23:02:31.097544 30816 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 23:02:31.084876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32986 (local address 127.25.94.62:34799)
0807 23:02:31.085073 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0807 23:02:31.085080 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 23:02:31.095039 (+ 9959us) server_negotiation.cc:407] Connection header received
0807 23:02:31.095363 (+ 324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:02:31.095370 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:02:31.095580 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:02:31.095723 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:02:31.097221 (+ 1498us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32986: BlockingRecv error: recv got EOF from 127.0.0.1:32986 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260807 23:02:31.099248 30683 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 23:02:31.084954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35400 (local address 127.25.94.61:38653)
0807 23:02:31.085133 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0807 23:02:31.085139 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 23:02:31.095754 (+ 10615us) server_negotiation.cc:407] Connection header received
0807 23:02:31.097343 (+ 1589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:02:31.097349 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:02:31.097443 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:02:31.097525 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:02:31.099010 (+ 1485us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35400: BlockingRecv error: recv got EOF from 127.0.0.1:35400 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260807 23:02:31.100574 30809 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 23:02:31.096367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46100 (local address 127.25.94.60:37557)
0807 23:02:31.096547 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0807 23:02:31.096553 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 23:02:31.098840 (+ 2287us) server_negotiation.cc:407] Connection header received
0807 23:02:31.099112 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:02:31.099118 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:02:31.099203 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:02:31.099277 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:02:31.100298 (+ 1021us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46100: BlockingRecv error: recv got EOF from 127.0.0.1:46100 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260807 23:02:31.109162 30914 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9bcfdd00153040c7ae21f38a2ffaf010: acquiring CA information for follower catalog manager: success
I20260807 23:02:31.112162 30914 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9bcfdd00153040c7ae21f38a2ffaf010: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260807 23:02:31.204280 30683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:31.176836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35402 (local address 127.25.94.61:38653)
0807 23:02:31.177360 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0807 23:02:31.177368 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 23:02:31.195441 (+ 18073us) server_negotiation.cc:407] Connection header received
0807 23:02:31.196089 (+ 648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:02:31.196118 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:02:31.196355 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:02:31.196510 (+ 155us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 23:02:31.197471 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:31.198687 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:31.200916 (+ 2229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:31.201660 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:31.201766 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:02:31.202646 (+ 880us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 23:02:31.203401 (+ 755us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 23:02:31.203530 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0807 23:02:31.203781 (+ 251us) server_negotiation.cc:299] Negotiation successful
0807 23:02:31.203889 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240}
W20260807 23:02:31.273087 30809 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 23:02:31.257593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46112 (local address 127.25.94.60:37557)
0807 23:02:31.257879 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0807 23:02:31.257889 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 23:02:31.271022 (+ 13133us) server_negotiation.cc:407] Connection header received
0807 23:02:31.271342 (+ 320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:02:31.271349 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:02:31.271518 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:02:31.271630 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:02:31.272842 (+ 1212us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46112: BlockingRecv error: recv got EOF from 127.0.0.1:46112 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
I20260807 23:02:31.321942 30683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:31.306789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35410 (local address 127.25.94.61:38653)
0807 23:02:31.307020 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0807 23:02:31.307029 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 23:02:31.314513 (+ 7484us) server_negotiation.cc:407] Connection header received
0807 23:02:31.315041 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:02:31.315049 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:02:31.315146 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:02:31.315217 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 23:02:31.316009 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:31.317248 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:31.319371 (+ 2123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:31.320146 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:31.320243 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:02:31.320949 (+ 706us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 23:02:31.321116 (+ 167us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 23:02:31.321225 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0807 23:02:31.321502 (+ 277us) server_negotiation.cc:299] Negotiation successful
0807 23:02:31.321594 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
W20260807 23:02:31.386059 30809 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 23:02:31.374754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46118 (local address 127.25.94.60:37557)
0807 23:02:31.375021 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0807 23:02:31.375028 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 23:02:31.384070 (+ 9042us) server_negotiation.cc:407] Connection header received
0807 23:02:31.384318 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:02:31.384324 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:02:31.384439 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:02:31.384560 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:02:31.385744 (+ 1184us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46118: BlockingRecv error: recv got EOF from 127.0.0.1:46118 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
I20260807 23:02:31.440578 30683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:31.424832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35426 (local address 127.25.94.61:38653)
0807 23:02:31.425036 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0807 23:02:31.425042 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 23:02:31.432582 (+ 7540us) server_negotiation.cc:407] Connection header received
0807 23:02:31.433234 (+ 652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:02:31.433240 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:02:31.433340 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:02:31.433418 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 23:02:31.434225 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:31.435396 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:31.438117 (+ 2721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:31.438762 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:31.438884 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:02:31.439571 (+ 687us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 23:02:31.439760 (+ 189us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 23:02:31.439844 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0807 23:02:31.440106 (+ 262us) server_negotiation.cc:299] Negotiation successful
0807 23:02:31.440213 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
W20260807 23:02:31.488181 30809 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 23:02:31.485434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46132 (local address 127.25.94.60:37557)
0807 23:02:31.485631 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0807 23:02:31.485637 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 23:02:31.486209 (+ 572us) server_negotiation.cc:407] Connection header received
0807 23:02:31.486570 (+ 361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:02:31.486576 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:02:31.486675 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:02:31.486762 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:02:31.487918 (+ 1156us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46132: BlockingRecv error: recv got EOF from 127.0.0.1:46132 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
I20260807 23:02:31.548350 30683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:31.531352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35434 (local address 127.25.94.61:38653)
0807 23:02:31.531534 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0807 23:02:31.531544 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 23:02:31.541123 (+ 9579us) server_negotiation.cc:407] Connection header received
0807 23:02:31.541730 (+ 607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:02:31.541736 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:02:31.541854 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:02:31.541932 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 23:02:31.542726 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:31.543816 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:31.545918 (+ 2102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:31.546666 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:31.546769 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:02:31.547448 (+ 679us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 23:02:31.547589 (+ 141us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 23:02:31.547649 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0807 23:02:31.547882 (+ 233us) server_negotiation.cc:299] Negotiation successful
0807 23:02:31.547988 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260807 23:02:31.582273 30988 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a6b04948f9c45ce99e9a420c6b2563b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1a6b04948f9c45ce99e9a420c6b2563b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a6b04948f9c45ce99e9a420c6b2563b" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 34799 } } peers { permanent_uuid: "9bcfdd00153040c7ae21f38a2ffaf010" member_type: VOTER last_known_addr { host: "127.25.94.61" port: 38653 } } peers { permanent_uuid: "3bdd8a084786405ca9faed30265c46be" member_type: VOTER last_known_addr { host: "127.25.94.60" port: 37557 } } }
I20260807 23:02:31.582901 30988 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a6b04948f9c45ce99e9a420c6b2563b [sys.catalog]: This master's current role is: LEADER
W20260807 23:02:31.599272 30809 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 23:02:31.590653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46136 (local address 127.25.94.60:37557)
0807 23:02:31.590900 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0807 23:02:31.590906 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 23:02:31.596749 (+ 5843us) server_negotiation.cc:407] Connection header received
0807 23:02:31.597328 (+ 579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:02:31.597335 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:02:31.597433 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:02:31.597515 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:02:31.599050 (+ 1535us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46136: BlockingRecv error: recv got EOF from 127.0.0.1:46136 (error 108)
Metrics: {"server-negotiator.queue_time_us":117}
I20260807 23:02:31.617774 30683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:31.611701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35438 (local address 127.25.94.61:38653)
0807 23:02:31.611944 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0807 23:02:31.611953 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 23:02:31.612543 (+ 590us) server_negotiation.cc:407] Connection header received
0807 23:02:31.612775 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:02:31.612780 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:02:31.612867 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:02:31.612984 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 23:02:31.613524 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:31.614357 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:31.615742 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:31.616208 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:31.616354 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:02:31.616806 (+ 452us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 23:02:31.616945 (+ 139us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 23:02:31.617046 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0807 23:02:31.617245 (+ 199us) server_negotiation.cc:299] Negotiation successful
0807 23:02:31.617330 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
W20260807 23:02:31.626439 30809 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 23:02:31.624603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46146 (local address 127.25.94.60:37557)
0807 23:02:31.624795 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0807 23:02:31.624801 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 23:02:31.624917 (+ 116us) server_negotiation.cc:407] Connection header received
0807 23:02:31.625216 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:02:31.625221 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:02:31.625307 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:02:31.625425 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:02:31.626190 (+ 765us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46146: BlockingRecv error: recv got EOF from 127.0.0.1:46146 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
I20260807 23:02:31.652107 30683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:31.646443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35454 (local address 127.25.94.61:38653)
0807 23:02:31.646689 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0807 23:02:31.646698 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 23:02:31.647275 (+ 577us) server_negotiation.cc:407] Connection header received
0807 23:02:31.647519 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:02:31.647524 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:02:31.647594 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:02:31.647695 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 23:02:31.648144 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:31.648904 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:31.650299 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:31.650729 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:31.650871 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:02:31.651309 (+ 438us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 23:02:31.651436 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 23:02:31.651547 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0807 23:02:31.651704 (+ 157us) server_negotiation.cc:299] Negotiation successful
0807 23:02:31.651785 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
W20260807 23:02:31.661916 30809 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 23:02:31.659408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46152 (local address 127.25.94.60:37557)
0807 23:02:31.659645 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0807 23:02:31.659654 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 23:02:31.660190 (+ 536us) server_negotiation.cc:407] Connection header received
0807 23:02:31.660460 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:02:31.660466 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:02:31.660559 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:02:31.660640 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:02:31.661683 (+ 1043us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46152: BlockingRecv error: recv got EOF from 127.0.0.1:46152 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
I20260807 23:02:31.704352 30683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:31.697903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35468 (local address 127.25.94.61:38653)
0807 23:02:31.698211 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0807 23:02:31.698218 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 23:02:31.698631 (+ 413us) server_negotiation.cc:407] Connection header received
0807 23:02:31.698933 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:02:31.698938 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:02:31.699017 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:02:31.699111 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 23:02:31.699630 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:31.700433 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:31.702395 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:31.703013 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:31.703117 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:02:31.703583 (+ 466us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 23:02:31.703684 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 23:02:31.703745 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0807 23:02:31.703904 (+ 159us) server_negotiation.cc:299] Negotiation successful
0807 23:02:31.703972 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
W20260807 23:02:31.712993 30809 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 23:02:31.710484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46168 (local address 127.25.94.60:37557)
0807 23:02:31.710737 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0807 23:02:31.710746 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 23:02:31.711497 (+ 751us) server_negotiation.cc:407] Connection header received
0807 23:02:31.711702 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:02:31.711706 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:02:31.711799 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:02:31.711884 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:02:31.712730 (+ 846us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46168: BlockingRecv error: recv got EOF from 127.0.0.1:46168 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
I20260807 23:02:31.786648 30683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:31.781074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35474 (local address 127.25.94.61:38653)
0807 23:02:31.781336 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0807 23:02:31.781343 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 23:02:31.781659 (+ 316us) server_negotiation.cc:407] Connection header received
0807 23:02:31.781903 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:02:31.781908 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:02:31.781974 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:02:31.782075 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 23:02:31.782539 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:31.783440 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:31.784914 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:31.785415 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:31.785552 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:02:31.785953 (+ 401us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 23:02:31.786049 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 23:02:31.786149 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0807 23:02:31.786300 (+ 151us) server_negotiation.cc:299] Negotiation successful
0807 23:02:31.786380 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
W20260807 23:02:31.794406 30809 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 23:02:31.792516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46182 (local address 127.25.94.60:37557)
0807 23:02:31.792791 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0807 23:02:31.792798 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 23:02:31.793048 (+ 250us) server_negotiation.cc:407] Connection header received
0807 23:02:31.793232 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:02:31.793238 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:02:31.793319 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:02:31.793410 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:02:31.794178 (+ 768us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46182: BlockingRecv error: recv got EOF from 127.0.0.1:46182 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
I20260807 23:02:31.931643 30683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:31.926473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35490 (local address 127.25.94.61:38653)
0807 23:02:31.926671 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0807 23:02:31.926676 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 23:02:31.927078 (+ 402us) server_negotiation.cc:407] Connection header received
0807 23:02:31.927237 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:02:31.927241 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:02:31.927312 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:02:31.927415 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 23:02:31.927855 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:31.928681 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:31.930151 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:31.930551 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:31.930645 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:02:31.931027 (+ 382us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 23:02:31.931126 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 23:02:31.931191 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0807 23:02:31.931365 (+ 174us) server_negotiation.cc:299] Negotiation successful
0807 23:02:31.931429 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
W20260807 23:02:31.939616 30809 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 23:02:31.937615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46192 (local address 127.25.94.60:37557)
0807 23:02:31.937824 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0807 23:02:31.937832 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 23:02:31.938243 (+ 411us) server_negotiation.cc:407] Connection header received
0807 23:02:31.938429 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:02:31.938433 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:02:31.938509 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:02:31.938618 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:02:31.939419 (+ 801us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46192: BlockingRecv error: recv got EOF from 127.0.0.1:46192 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260807 23:02:31.964920 30882 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3bdd8a084786405ca9faed30265c46be: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260807 23:02:32.205397 30683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:32.200018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35498 (local address 127.25.94.61:38653)
0807 23:02:32.200262 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0807 23:02:32.200269 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 23:02:32.200525 (+ 256us) server_negotiation.cc:407] Connection header received
0807 23:02:32.200692 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:02:32.200697 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:02:32.200769 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:02:32.200876 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 23:02:32.201468 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:32.202201 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:32.203661 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:32.204110 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:32.204217 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:02:32.204623 (+ 406us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 23:02:32.204747 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 23:02:32.204817 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0807 23:02:32.204957 (+ 140us) server_negotiation.cc:299] Negotiation successful
0807 23:02:32.205025 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260807 23:02:32.217551 30809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:32.211541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46194 (local address 127.25.94.60:37557)
0807 23:02:32.211782 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0807 23:02:32.211790 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 23:02:32.212073 (+ 283us) server_negotiation.cc:407] Connection header received
0807 23:02:32.212298 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:02:32.212302 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:02:32.212460 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:02:32.212586 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 23:02:32.213006 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:32.213790 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:32.215180 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:32.215587 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:32.215729 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:02:32.216268 (+ 539us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 23:02:32.216868 (+ 600us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 23:02:32.217046 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0807 23:02:32.217132 (+ 86us) server_negotiation.cc:299] Negotiation successful
0807 23:02:32.217270 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260807 23:02:32.232014 31105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:32.224276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32994 (local address 127.25.94.62:34799)
0807 23:02:32.225131 (+ 855us) server_negotiation.cc:206] Beginning negotiation
0807 23:02:32.225138 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 23:02:32.225158 (+ 20us) server_negotiation.cc:407] Connection header received
0807 23:02:32.225374 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:02:32.225378 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:02:32.225591 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:02:32.225789 (+ 198us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 23:02:32.226536 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:32.227667 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:32.229598 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:32.230265 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:32.230389 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:02:32.230918 (+ 529us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 23:02:32.231273 (+ 355us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 23:02:32.231415 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0807 23:02:32.231566 (+ 151us) server_negotiation.cc:299] Negotiation successful
0807 23:02:32.231659 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":729,"thread_start_us":620,"threads_started":1}
I20260807 23:02:32.233263 30683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:32.224354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35510 (local address 127.25.94.61:38653)
0807 23:02:32.224625 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0807 23:02:32.224635 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 23:02:32.225625 (+ 990us) server_negotiation.cc:407] Connection header received
0807 23:02:32.225977 (+ 352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:02:32.225986 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:02:32.226107 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:02:32.226186 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 23:02:32.227133 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:32.228156 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:32.230431 (+ 2275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:32.231226 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:32.231386 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:02:32.232375 (+ 989us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 23:02:32.232521 (+ 146us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 23:02:32.232608 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0807 23:02:32.232784 (+ 176us) server_negotiation.cc:299] Negotiation successful
0807 23:02:32.232869 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260807 23:02:32.233515 30809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:32.225752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46198 (local address 127.25.94.60:37557)
0807 23:02:32.225945 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0807 23:02:32.225951 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 23:02:32.227132 (+ 1181us) server_negotiation.cc:407] Connection header received
0807 23:02:32.227405 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:02:32.227411 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:02:32.227500 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:02:32.227599 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 23:02:32.228242 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:32.229404 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:32.231450 (+ 2046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:32.231986 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:32.232131 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:02:32.232684 (+ 553us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 23:02:32.232813 (+ 129us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 23:02:32.232935 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0807 23:02:32.233099 (+ 164us) server_negotiation.cc:299] Negotiation successful
0807 23:02:32.233228 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260807 23:02:32.238716 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 30436
I20260807 23:02:32.240211 30545 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 23:02:32.352288 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 30436
I20260807 23:02:32.379963 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 30564
I20260807 23:02:32.381045 30669 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 23:02:32.492811 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 30564
I20260807 23:02:32.521821 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 30686
I20260807 23:02:32.523000 30793 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 23:02:32.634042 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 30686
2026-08-07T23:02:32Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (5259 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786143361779964-25976-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 23:02:32 dist-test-slave-z8x9 krb5kdc[31120](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 23:02:32 dist-test-slave-z8x9 krb5kdc[31120](info): set up 2 sockets
Aug 07 23:02:32 dist-test-slave-z8x9 krb5kdc[31120](info): commencing operation
krb5kdc: starting...
W20260807 23:02:34.027319 25976 mini_kdc.cc:121] Time spent starting KDC: real 1.339s user 0.000s sys 0.049s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786143361779964-25976-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786143361779964-25976-0/krb5kdc/test-user.keytab.
Aug 07 23:02:34 dist-test-slave-z8x9 krb5kdc[31120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143754, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T23:02:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T23:02:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.94.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.62.keytab.
Entry for principal kudu/127.25.94.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.62.keytab.
I20260807 23:02:34.388613 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786143361779964-25976-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786143361779964-25976-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786143361779964-25976-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.62:40737
--webserver_interface=127.25.94.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.62.keytab
--principal=kudu/127.25.94.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.20:41141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.94.62:40737,127.25.94.61:37133,127.25.94.60:36589
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786143361779964-25976-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786143361779964-25976-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786143361779964-25976-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 23:02:34.746536 31139 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260807 23:02:34.746868 31139 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 23:02:34.746950 31139 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 23:02:34.747032 31139 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 23:02:34.754326 31139 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 23:02:34.754421 31139 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 23:02:34.754479 31139 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 23:02:34.754534 31139 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 23:02:34.754596 31139 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 23:02:34.764542 31139 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:41141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786143361779964-25976-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786143361779964-25976-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.94.62:40737,127.25.94.61:37133,127.25.94.60:36589
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.62.keytab
--principal=kudu/127.25.94.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.62:40737
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.94.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786143361779964-25976-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 23:02:34.765841 31139 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 23:02:34.767123 31139 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 23:02:34.778972 31145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 23:02:34.779284 31146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 23:02:34.779121 31148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 23:02:34.780823 31139 server_base.cc:1034] running on GCE node
I20260807 23:02:34.781701 31139 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 23:02:34.783135 31139 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 23:02:34.784395 31139 hybrid_clock.cc:648] HybridClock initialized: now 1786143754784338 us; error 29 us; skew 500 ppm
I20260807 23:02:34.784770 31139 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 23:02:34 dist-test-slave-z8x9 krb5kdc[31120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143754, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 23:02:34.788038 31139 init.cc:377] Logged in from keytab as kudu/127.25.94.62@KRBTEST.COM (short username kudu)
I20260807 23:02:34.789538 31139 webserver.cc:460] Webserver started at http://127.25.94.62:34537/ using document root <none> and password file <none>
I20260807 23:02:34.790241 31139 fs_manager.cc:359] Metadata directory not provided
I20260807 23:02:34.790359 31139 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 23:02:34.790688 31139 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 23:02:34.793748 31139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786143361779964-25976-0/minicluster-data/master-0/data/instance:
uuid: "c8ef131160eb445ba860c4129128a5fb"
format_stamp: "Formatted at 2026-08-07 23:02:34 on dist-test-slave-z8x9"
server_key: "32c8dd8f5c1d7788e13071e2e3ab6687"
server_key_iv: "cd9c1ca3cdbd8e08422471604e7fed15"
server_key_version: "encryptionkey@0"
I20260807 23:02:34.794567 31139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786143361779964-25976-0/minicluster-data/master-0/wal/instance:
uuid: "c8ef131160eb445ba860c4129128a5fb"
format_stamp: "Formatted at 2026-08-07 23:02:34 on dist-test-slave-z8x9"
server_key: "32c8dd8f5c1d7788e13071e2e3ab6687"
server_key_iv: "cd9c1ca3cdbd8e08422471604e7fed15"
server_key_version: "encryptionkey@0"
I20260807 23:02:34.799917 31139 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260807 23:02:34.803427 31160 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 23:02:34.804775 31139 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 23:02:34.804968 31139 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786143361779964-25976-0/minicluster-data/master-0/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786143361779964-25976-0/minicluster-data/master-0/wal
uuid: "c8ef131160eb445ba860c4129128a5fb"
format_stamp: "Formatted at 2026-08-07 23:02:34 on dist-test-slave-z8x9"
server_key: "32c8dd8f5c1d7788e13071e2e3ab6687"
server_key_iv: "cd9c1ca3cdbd8e08422471604e7fed15"
server_key_version: "encryptionkey@0"
I20260807 23:02:34.805227 31139 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786143361779964-25976-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786143361779964-25976-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786143361779964-25976-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 23:02:34.826808 31139 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 23:02:34.830974 31139 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 23:02:34.831255 31139 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 23:02:34.850872 31139 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.62:40737
I20260807 23:02:34.850914 31234 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.62:40737 every 8 connection(s)
I20260807 23:02:34.852238 31139 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786143361779964-25976-0/minicluster-data/master-0/data/info.pb
I20260807 23:02:34.853992 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 31139
I20260807 23:02:34.854192 25976 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786143361779964-25976-0/minicluster-data/master-0/wal/instance
I20260807 23:02:34.854789 25976 external_mini_cluster.cc:1428] Setting key 18e2f7a576375da2cb1a5bc8c9814cad
I20260807 23:02:34.856886 31235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 23:02:34.866900 31235 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.94.62" port: 40737 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 23:02:34 dist-test-slave-z8x9 krb5kdc[31120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143754, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.62@KRBTEST.COM for kudu/127.25.94.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.25.94.61@KRBTEST.COM; defaulting to no policy
I20260807 23:02:34.893904 31239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:34.871004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51832 (local address 127.25.94.62:40737)
0807 23:02:34.872128 (+ 1124us) server_negotiation.cc:206] Beginning negotiation
0807 23:02:34.872146 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0807 23:02:34.872365 (+ 219us) server_negotiation.cc:407] Connection header received
0807 23:02:34.874514 (+ 2149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:02:34.874520 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:02:34.874769 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:02:34.874972 (+ 203us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:02:34.876432 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:34.878573 (+ 2141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:34.880055 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:34.881155 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:34.881398 (+ 243us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:02:34.884946 (+ 3548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:02:34.884959 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:02:34.884980 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 23:02:34.885023 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:02:34.889405 (+ 4382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:02:34.890145 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:02:34.890156 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:02:34.890161 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:02:34.890383 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:02:34.890808 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:02:34.890815 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:02:34.890820 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:02:34.891599 (+ 779us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:02:34.891802 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0807 23:02:34.892413 (+ 611us) server_negotiation.cc:299] Negotiation successful
0807 23:02:34.892685 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":922,"spinlock_wait_cycles":15744,"thread_start_us":552,"threads_started":1}
I20260807 23:02:34.893923 31238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:34.871004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:40737 (local address 127.0.0.1:51832)
0807 23:02:34.872197 (+ 1193us) negotiation.cc:107] Waiting for socket to connect
0807 23:02:34.872242 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0807 23:02:34.874103 (+ 1861us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 23:02:34.875019 (+ 916us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 23:02:34.875028 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 23:02:34.875766 (+ 738us) client_negotiation.cc:189] Negotiated authn=SASL
0807 23:02:34.876120 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:02:34.876138 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:02:34.878766 (+ 2628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:02:34.878771 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:02:34.879876 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:02:34.879889 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:02:34.881424 (+ 1535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:02:34.881429 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:02:34.881564 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:02:34.882316 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 23:02:34.882340 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0807 23:02:34.884691 (+ 2351us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 23:02:34.889554 (+ 4863us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 23:02:34.889566 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 23:02:34.889577 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0807 23:02:34.889997 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 23:02:34.890509 (+ 512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 23:02:34.890512 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 23:02:34.890514 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 23:02:34.890677 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 23:02:34.891764 (+ 1087us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 23:02:34.891770 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 23:02:34.892027 (+ 257us) client_negotiation.cc:769] Sending connection context
0807 23:02:34.892272 (+ 245us) client_negotiation.cc:240] Negotiation successful
0807 23:02:34.892849 (+ 577us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":867,"mutex_wait_us":164,"thread_start_us":549,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.61@KRBTEST.COM" created.
I20260807 23:02:34.897120 31235 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.94.61" port: 37133 } has no permanent_uuid. Determining permanent_uuid...
I20260807 23:02:34.898509 31238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:34.897964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.61:37133 (local address 127.0.0.1:49502)
0807 23:02:34.898187 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0807 23:02:34.898333 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.61:37133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260807 23:02:34.898847 31165 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.94.61:37133: connect: Connection refused (error 111)
I20260807 23:02:34.900535 31238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:34.900078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.61:37133 (local address 127.0.0.1:49510)
0807 23:02:34.900270 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0807 23:02:34.900362 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.61:37133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260807 23:02:34.900820 31235 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.94.61:37133: Network error: Client connection negotiation failed: client connection to 127.25.94.61:37133: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.61.keytab.
Entry for principal kudu/127.25.94.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.61.keytab.
I20260807 23:02:34.934715 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786143361779964-25976-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786143361779964-25976-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786143361779964-25976-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786143361779964-25976-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.61:37133
--webserver_interface=127.25.94.61
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.61.keytab
--principal=kudu/127.25.94.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.20:41141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.94.62:40737,127.25.94.61:37133,127.25.94.60:36589
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786143361779964-25976-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786143361779964-25976-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786143361779964-25976-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 23:02:34.950203 31235 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.94.61" port: 37133 } attempt: 1
I20260807 23:02:34.951502 31238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:34.951008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.61:37133 (local address 127.0.0.1:49512)
0807 23:02:34.951247 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0807 23:02:34.951357 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.61:37133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260807 23:02:34.952654 31238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:34.952253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.61:37133 (local address 127.0.0.1:49528)
0807 23:02:34.952407 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0807 23:02:34.952492 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.61:37133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260807 23:02:34.952847 31235 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.94.61:37133: Network error: Client connection negotiation failed: client connection to 127.25.94.61:37133: connect: Connection refused (error 111)
I20260807 23:02:35.021111 31235 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.94.61" port: 37133 } attempt: 2
I20260807 23:02:35.022441 31238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:35.021977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.61:37133 (local address 127.0.0.1:49530)
0807 23:02:35.022187 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0807 23:02:35.022292 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.61:37133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260807 23:02:35.023572 31238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:35.023157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.61:37133 (local address 127.0.0.1:49532)
0807 23:02:35.023355 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0807 23:02:35.023440 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.61:37133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
W20260807 23:02:35.023800 31235 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.94.61:37133: Network error: Client connection negotiation failed: client connection to 127.25.94.61:37133: connect: Connection refused (error 111)
I20260807 23:02:35.115067 31235 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.94.61" port: 37133 } attempt: 3
I20260807 23:02:35.116346 31238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:35.115855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.61:37133 (local address 127.0.0.1:49538)
0807 23:02:35.116065 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0807 23:02:35.116162 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.61:37133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260807 23:02:35.117472 31238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:35.117048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.61:37133 (local address 127.0.0.1:49552)
0807 23:02:35.117249 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0807 23:02:35.117332 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.61:37133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260807 23:02:35.117666 31235 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.94.61:37133: Network error: Client connection negotiation failed: client connection to 127.25.94.61:37133: connect: Connection refused (error 111)
I20260807 23:02:35.260931 31235 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.94.61" port: 37133 } attempt: 4
I20260807 23:02:35.262461 31238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:35.261671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.61:37133 (local address 127.0.0.1:49564)
0807 23:02:35.262068 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0807 23:02:35.262261 (+ 193us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.61:37133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210}
I20260807 23:02:35.263705 31238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:35.263256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.61:37133 (local address 127.0.0.1:49572)
0807 23:02:35.263441 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0807 23:02:35.263541 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.61:37133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260807 23:02:35.263963 31235 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.94.61:37133: Network error: Client connection negotiation failed: client connection to 127.25.94.61:37133: connect: Connection refused (error 111)
W20260807 23:02:35.292614 31242 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260807 23:02:35.292974 31242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 23:02:35.293048 31242 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 23:02:35.293133 31242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 23:02:35.300767 31242 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 23:02:35.300882 31242 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 23:02:35.300941 31242 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 23:02:35.300987 31242 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 23:02:35.301050 31242 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 23:02:35.311890 31242 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:41141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786143361779964-25976-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786143361779964-25976-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.25.94.62:40737,127.25.94.61:37133,127.25.94.60:36589
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.61.keytab
--principal=kudu/127.25.94.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.61:37133
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786143361779964-25976-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.25.94.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786143361779964-25976-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 23:02:35.313745 31242 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 23:02:35.315058 31242 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 23:02:35.327474 31254 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 23:02:35.327579 31258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 23:02:35.327600 31255 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 23:02:35.328791 31242 server_base.cc:1034] running on GCE node
I20260807 23:02:35.329638 31242 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 23:02:35.331149 31242 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 23:02:35.332398 31242 hybrid_clock.cc:648] HybridClock initialized: now 1786143755332313 us; error 50 us; skew 500 ppm
I20260807 23:02:35.332772 31242 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 23:02:35 dist-test-slave-z8x9 krb5kdc[31120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143755, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 23:02:35.341342 31242 init.cc:377] Logged in from keytab as kudu/127.25.94.61@KRBTEST.COM (short username kudu)
I20260807 23:02:35.342810 31242 webserver.cc:460] Webserver started at http://127.25.94.61:40545/ using document root <none> and password file <none>
I20260807 23:02:35.343501 31242 fs_manager.cc:359] Metadata directory not provided
I20260807 23:02:35.343597 31242 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 23:02:35.343957 31242 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 23:02:35.347055 31242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786143361779964-25976-0/minicluster-data/master-1/data/instance:
uuid: "b560dd8a492f49609945578af494f2ab"
format_stamp: "Formatted at 2026-08-07 23:02:35 on dist-test-slave-z8x9"
server_key: "d43f70bb3728d81b047ed28f7a4b81df"
server_key_iv: "cb8e1c08d64769b88f864c35a038f7fd"
server_key_version: "encryptionkey@0"
I20260807 23:02:35.347885 31242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786143361779964-25976-0/minicluster-data/master-1/wal/instance:
uuid: "b560dd8a492f49609945578af494f2ab"
format_stamp: "Formatted at 2026-08-07 23:02:35 on dist-test-slave-z8x9"
server_key: "d43f70bb3728d81b047ed28f7a4b81df"
server_key_iv: "cb8e1c08d64769b88f864c35a038f7fd"
server_key_version: "encryptionkey@0"
I20260807 23:02:35.353547 31242 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260807 23:02:35.357064 31266 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 23:02:35.358646 31242 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 23:02:35.358831 31242 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786143361779964-25976-0/minicluster-data/master-1/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786143361779964-25976-0/minicluster-data/master-1/wal
uuid: "b560dd8a492f49609945578af494f2ab"
format_stamp: "Formatted at 2026-08-07 23:02:35 on dist-test-slave-z8x9"
server_key: "d43f70bb3728d81b047ed28f7a4b81df"
server_key_iv: "cb8e1c08d64769b88f864c35a038f7fd"
server_key_version: "encryptionkey@0"
I20260807 23:02:35.359088 31242 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786143361779964-25976-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786143361779964-25976-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786143361779964-25976-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 23:02:35.371609 31242 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 23:02:35.375741 31242 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 23:02:35.376024 31242 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 23:02:35.396733 31242 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.61:37133
I20260807 23:02:35.396757 31340 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.61:37133 every 8 connection(s)
I20260807 23:02:35.398379 31242 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786143361779964-25976-0/minicluster-data/master-1/data/info.pb
I20260807 23:02:35.400259 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 31242
I20260807 23:02:35.400470 25976 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786143361779964-25976-0/minicluster-data/master-1/wal/instance
I20260807 23:02:35.401070 25976 external_mini_cluster.cc:1428] Setting key fe155a911d02f2312e54f8a55061abf5
I20260807 23:02:35.403265 31341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 23:02:35.413843 31341 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.94.62" port: 40737 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 23:02:35 dist-test-slave-z8x9 krb5kdc[31120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143755, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.61@KRBTEST.COM for kudu/127.25.94.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.25.94.60@KRBTEST.COM; defaulting to no policy
I20260807 23:02:35.440743 31344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:35.417395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51834 (local address 127.25.94.62:40737)
0807 23:02:35.417937 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0807 23:02:35.417943 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 23:02:35.419009 (+ 1066us) server_negotiation.cc:407] Connection header received
0807 23:02:35.422093 (+ 3084us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:02:35.422098 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:02:35.422216 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:02:35.422318 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:02:35.424637 (+ 2319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:35.425469 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:35.428080 (+ 2611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:35.428539 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:35.428648 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:02:35.433397 (+ 4749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:02:35.433403 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:02:35.433406 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 23:02:35.433456 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:02:35.437436 (+ 3980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:02:35.438277 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:02:35.438285 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:02:35.438289 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:02:35.438460 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:02:35.438924 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:02:35.438930 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:02:35.438933 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:02:35.439380 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:02:35.439552 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0807 23:02:35.440209 (+ 657us) server_negotiation.cc:299] Negotiation successful
0807 23:02:35.440460 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":268,"threads_started":1}
I20260807 23:02:35.441921 31345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:35.417936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:40737 (local address 127.0.0.1:51834)
0807 23:02:35.418779 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0807 23:02:35.418860 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0807 23:02:35.421743 (+ 2883us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 23:02:35.422619 (+ 876us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 23:02:35.422632 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 23:02:35.423478 (+ 846us) client_negotiation.cc:189] Negotiated authn=SASL
0807 23:02:35.424377 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:02:35.424406 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:02:35.425660 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:02:35.425669 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:02:35.427890 (+ 2221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:02:35.427908 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:02:35.428826 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:02:35.428840 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:02:35.429236 (+ 396us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:02:35.430067 (+ 831us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 23:02:35.430109 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0807 23:02:35.433030 (+ 2921us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 23:02:35.437622 (+ 4592us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 23:02:35.437630 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 23:02:35.437655 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0807 23:02:35.438120 (+ 465us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 23:02:35.438592 (+ 472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 23:02:35.438595 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 23:02:35.438597 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 23:02:35.438796 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 23:02:35.439509 (+ 713us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 23:02:35.439515 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 23:02:35.439877 (+ 362us) client_negotiation.cc:769] Sending connection context
0807 23:02:35.440120 (+ 243us) client_negotiation.cc:240] Negotiation successful
0807 23:02:35.440648 (+ 528us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":302,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.94.60@KRBTEST.COM" created.
I20260807 23:02:35.443840 31341 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.94.61" port: 37133 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 23:02:35 dist-test-slave-z8x9 krb5kdc[31120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143755, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.61@KRBTEST.COM for kudu/127.25.94.61@KRBTEST.COM
I20260807 23:02:35.459256 31345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:35.444741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.61:37133 (local address 127.0.0.1:49586)
0807 23:02:35.444951 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0807 23:02:35.444977 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0807 23:02:35.445061 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 23:02:35.446091 (+ 1030us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 23:02:35.446096 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 23:02:35.446586 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0807 23:02:35.446717 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:02:35.446725 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:02:35.448107 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:02:35.448114 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:02:35.449080 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:02:35.449093 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:02:35.450049 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:02:35.450054 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:02:35.450310 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:02:35.450959 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 23:02:35.450976 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 23:02:35.453064 (+ 2088us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 23:02:35.456476 (+ 3412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 23:02:35.456485 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 23:02:35.456490 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 23:02:35.456903 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 23:02:35.457382 (+ 479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 23:02:35.457386 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 23:02:35.457390 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 23:02:35.457519 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 23:02:35.458175 (+ 656us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 23:02:35.458179 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 23:02:35.458316 (+ 137us) client_negotiation.cc:769] Sending connection context
0807 23:02:35.458496 (+ 180us) client_negotiation.cc:240] Negotiation successful
0807 23:02:35.458810 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56,"mutex_wait_us":43}
I20260807 23:02:35.459313 31346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:35.445002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49586 (local address 127.25.94.61:37133)
0807 23:02:35.445671 (+ 669us) server_negotiation.cc:206] Beginning negotiation
0807 23:02:35.445677 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 23:02:35.445712 (+ 35us) server_negotiation.cc:407] Connection header received
0807 23:02:35.445756 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:02:35.445762 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:02:35.445941 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:02:35.446099 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:02:35.446897 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:35.447966 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:35.449343 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:35.449900 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:35.450040 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:02:35.453333 (+ 3293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:02:35.453345 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:02:35.453360 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 23:02:35.453389 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:02:35.456314 (+ 2925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:02:35.457046 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:02:35.457061 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:02:35.457063 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:02:35.457255 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:02:35.457637 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:02:35.457642 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:02:35.457645 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:02:35.458018 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:02:35.458233 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0807 23:02:35.458657 (+ 424us) server_negotiation.cc:299] Negotiation successful
0807 23:02:35.458924 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":41,"server-negotiator.queue_time_us":528,"thread_start_us":391,"threads_started":1}
I20260807 23:02:35.462206 31341 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.94.60" port: 36589 } has no permanent_uuid. Determining permanent_uuid...
I20260807 23:02:35.463835 31345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:35.463207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.60:36589 (local address 127.0.0.1:52928)
0807 23:02:35.463436 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0807 23:02:35.463597 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.60:36589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260807 23:02:35.464156 31273 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.94.60:36589: connect: Connection refused (error 111)
I20260807 23:02:35.466091 31345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:35.465612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.60:36589 (local address 127.0.0.1:52934)
0807 23:02:35.465808 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0807 23:02:35.465931 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.60:36589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260807 23:02:35.466362 31341 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.94.60:36589: Network error: Client connection negotiation failed: client connection to 127.25.94.60:36589: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.94.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.60.keytab.
Entry for principal kudu/127.25.94.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.60.keytab.
I20260807 23:02:35.485724 25976 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknHbun_/build/asan/bin/kudu
/tmp/dist-test-tasknHbun_/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786143361779964-25976-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786143361779964-25976-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786143361779964-25976-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786143361779964-25976-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.60:36589
--webserver_interface=127.25.94.60
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.60.keytab
--principal=kudu/127.25.94.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.94.20:41141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.94.62:40737,127.25.94.61:37133,127.25.94.60:36589
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786143361779964-25976-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786143361779964-25976-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786143361779964-25976-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 23:02:35.515754 31341 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.94.60" port: 36589 } attempt: 1
I20260807 23:02:35.517241 31345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:35.516538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.60:36589 (local address 127.0.0.1:52938)
0807 23:02:35.516793 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0807 23:02:35.516941 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.60:36589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260807 23:02:35.518399 31345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:35.517920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.60:36589 (local address 127.0.0.1:52948)
0807 23:02:35.518122 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0807 23:02:35.518228 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.60:36589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260807 23:02:35.518664 31341 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.94.60:36589: Network error: Client connection negotiation failed: client connection to 127.25.94.60:36589: connect: Connection refused (error 111)
I20260807 23:02:35.563300 31235 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.94.61" port: 37133 } attempt: 5
Aug 07 23:02:35 dist-test-slave-z8x9 krb5kdc[31120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143754, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.62@KRBTEST.COM for kudu/127.25.94.61@KRBTEST.COM
I20260807 23:02:35.577813 31346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:35.564394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49588 (local address 127.25.94.61:37133)
0807 23:02:35.564619 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0807 23:02:35.564627 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 23:02:35.564663 (+ 36us) server_negotiation.cc:407] Connection header received
0807 23:02:35.564729 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:02:35.564734 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:02:35.564804 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:02:35.564914 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:02:35.565833 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:35.566566 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:35.567779 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:35.568213 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:35.568351 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:02:35.571947 (+ 3596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:02:35.571952 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:02:35.571955 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 23:02:35.571988 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:02:35.574841 (+ 2853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:02:35.575785 (+ 944us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:02:35.575790 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:02:35.575792 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:02:35.575914 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:02:35.576400 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:02:35.576403 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:02:35.576405 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:02:35.576680 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:02:35.576803 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0807 23:02:35.577395 (+ 592us) server_negotiation.cc:299] Negotiation successful
0807 23:02:35.577575 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260807 23:02:35.577957 31238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:35.564174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.61:37133 (local address 127.0.0.1:49588)
0807 23:02:35.564384 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0807 23:02:35.564405 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 23:02:35.564566 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 23:02:35.564969 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 23:02:35.564974 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 23:02:35.565487 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0807 23:02:35.565669 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:02:35.565679 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:02:35.566729 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:02:35.566735 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:02:35.567620 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:02:35.567632 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:02:35.568368 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:02:35.568374 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:02:35.568608 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:02:35.569356 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 23:02:35.569374 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 23:02:35.571709 (+ 2335us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 23:02:35.574991 (+ 3282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 23:02:35.574998 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 23:02:35.575006 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0807 23:02:35.575638 (+ 632us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 23:02:35.576044 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 23:02:35.576052 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 23:02:35.576058 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 23:02:35.576276 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 23:02:35.576803 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 23:02:35.576809 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 23:02:35.576988 (+ 179us) client_negotiation.cc:769] Sending connection context
0807 23:02:35.577290 (+ 302us) client_negotiation.cc:240] Negotiation successful
0807 23:02:35.577608 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74}
I20260807 23:02:35.579071 31235 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.94.60" port: 36589 } has no permanent_uuid. Determining permanent_uuid...
I20260807 23:02:35.580292 31238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:35.579868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.60:36589 (local address 127.0.0.1:52962)
0807 23:02:35.580042 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0807 23:02:35.580118 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.60:36589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260807 23:02:35.581499 31238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:35.581008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.60:36589 (local address 127.0.0.1:52972)
0807 23:02:35.581314 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0807 23:02:35.581386 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.60:36589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260807 23:02:35.581773 31235 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.94.60:36589: Network error: Client connection negotiation failed: client connection to 127.25.94.60:36589: connect: Connection refused (error 111)
I20260807 23:02:35.586922 31341 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.94.60" port: 36589 } attempt: 2
I20260807 23:02:35.588071 31345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:35.587574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.60:36589 (local address 127.0.0.1:52976)
0807 23:02:35.587783 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0807 23:02:35.587895 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.60:36589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260807 23:02:35.589087 31345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:35.588705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.60:36589 (local address 127.0.0.1:52982)
0807 23:02:35.588864 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0807 23:02:35.588950 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.60:36589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260807 23:02:35.589366 31341 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.94.60:36589: Network error: Client connection negotiation failed: client connection to 127.25.94.60:36589: connect: Connection refused (error 111)
I20260807 23:02:35.633019 31235 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.94.60" port: 36589 } attempt: 1
I20260807 23:02:35.634295 31238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:35.633860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.60:36589 (local address 127.0.0.1:52994)
0807 23:02:35.634039 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0807 23:02:35.634129 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.60:36589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260807 23:02:35.635345 31238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:35.635011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.60:36589 (local address 127.0.0.1:52998)
0807 23:02:35.635161 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0807 23:02:35.635236 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.60:36589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260807 23:02:35.635560 31235 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.94.60:36589: Network error: Client connection negotiation failed: client connection to 127.25.94.60:36589: connect: Connection refused (error 111)
I20260807 23:02:35.680615 31341 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.94.60" port: 36589 } attempt: 3
I20260807 23:02:35.682020 31345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:35.681397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.60:36589 (local address 127.0.0.1:53010)
0807 23:02:35.681670 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0807 23:02:35.681811 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.60:36589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260807 23:02:35.683290 31345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:35.682702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.60:36589 (local address 127.0.0.1:53022)
0807 23:02:35.683000 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0807 23:02:35.683117 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.60:36589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162}
W20260807 23:02:35.683545 31341 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.94.60:36589: Network error: Client connection negotiation failed: client connection to 127.25.94.60:36589: connect: Connection refused (error 111)
I20260807 23:02:35.703776 31235 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.94.60" port: 36589 } attempt: 2
I20260807 23:02:35.705006 31238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:35.704495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.60:36589 (local address 127.0.0.1:53038)
0807 23:02:35.704733 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0807 23:02:35.704843 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.60:36589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260807 23:02:35.706080 31238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:35.705723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.60:36589 (local address 127.0.0.1:53046)
0807 23:02:35.705911 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0807 23:02:35.705987 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.60:36589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260807 23:02:35.706292 31235 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.94.60:36589: Network error: Client connection negotiation failed: client connection to 127.25.94.60:36589: connect: Connection refused (error 111)
I20260807 23:02:35.812510 31235 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.94.60" port: 36589 } attempt: 3
I20260807 23:02:35.813614 31238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:35.813115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.60:36589 (local address 127.0.0.1:53060)
0807 23:02:35.813368 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0807 23:02:35.813455 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.60:36589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
I20260807 23:02:35.814697 31238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:35.814310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.60:36589 (local address 127.0.0.1:53074)
0807 23:02:35.814523 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0807 23:02:35.814592 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.60:36589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
W20260807 23:02:35.814891 31235 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.94.60:36589: Network error: Client connection negotiation failed: client connection to 127.25.94.60:36589: connect: Connection refused (error 111)
I20260807 23:02:35.826804 31341 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.94.60" port: 36589 } attempt: 4
I20260807 23:02:35.828087 31345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:35.827529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.60:36589 (local address 127.0.0.1:53082)
0807 23:02:35.827783 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0807 23:02:35.827890 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.60:36589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260807 23:02:35.829147 31345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:35.828833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.60:36589 (local address 127.0.0.1:53094)
0807 23:02:35.828968 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0807 23:02:35.829039 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.94.60:36589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260807 23:02:35.829492 31341 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.94.60:36589: Network error: Client connection negotiation failed: client connection to 127.25.94.60:36589: connect: Connection refused (error 111)
W20260807 23:02:35.837888 31351 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260807 23:02:35.838266 31351 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 23:02:35.838364 31351 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 23:02:35.838501 31351 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 23:02:35.845921 31351 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 23:02:35.846015 31351 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 23:02:35.846130 31351 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 23:02:35.846210 31351 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 23:02:35.846305 31351 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 23:02:35.856362 31351 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.94.20:41141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786143361779964-25976-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786143361779964-25976-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.25.94.62:40737,127.25.94.61:37133,127.25.94.60:36589
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786143361779964-25976-0/krb5kdc/kudu_127.25.94.60.keytab
--principal=kudu/127.25.94.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.94.60:36589
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786143361779964-25976-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.25.94.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786143361779964-25976-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 23:02:35.857656 31351 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 23:02:35.858929 31351 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 23:02:35.870535 31363 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 23:02:35.870878 31371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 23:02:35.870566 31359 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 23:02:35.871901 31351 server_base.cc:1034] running on GCE node
I20260807 23:02:35.872681 31351 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 23:02:35.874123 31351 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 23:02:35.875383 31351 hybrid_clock.cc:648] HybridClock initialized: now 1786143755875313 us; error 45 us; skew 500 ppm
I20260807 23:02:35.875726 31351 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 23:02:35 dist-test-slave-z8x9 krb5kdc[31120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143755, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 23:02:35.879120 31351 init.cc:377] Logged in from keytab as kudu/127.25.94.60@KRBTEST.COM (short username kudu)
I20260807 23:02:35.880618 31351 webserver.cc:460] Webserver started at http://127.25.94.60:45309/ using document root <none> and password file <none>
I20260807 23:02:35.881383 31351 fs_manager.cc:359] Metadata directory not provided
I20260807 23:02:35.881518 31351 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 23:02:35.881850 31351 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 23:02:35.884775 31351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786143361779964-25976-0/minicluster-data/master-2/data/instance:
uuid: "85e4774ac2f743048053be25c8281bd8"
format_stamp: "Formatted at 2026-08-07 23:02:35 on dist-test-slave-z8x9"
server_key: "e7921a8649ec724057c9bbc103ab295a"
server_key_iv: "7f0b64a45efdadcbfd867160ce0cc309"
server_key_version: "encryptionkey@0"
I20260807 23:02:35.885618 31351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786143361779964-25976-0/minicluster-data/master-2/wal/instance:
uuid: "85e4774ac2f743048053be25c8281bd8"
format_stamp: "Formatted at 2026-08-07 23:02:35 on dist-test-slave-z8x9"
server_key: "e7921a8649ec724057c9bbc103ab295a"
server_key_iv: "7f0b64a45efdadcbfd867160ce0cc309"
server_key_version: "encryptionkey@0"
I20260807 23:02:35.890842 31351 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260807 23:02:35.894459 31379 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 23:02:35.895787 31351 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 23:02:35.896080 31351 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786143361779964-25976-0/minicluster-data/master-2/data,/tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786143361779964-25976-0/minicluster-data/master-2/wal
uuid: "85e4774ac2f743048053be25c8281bd8"
format_stamp: "Formatted at 2026-08-07 23:02:35 on dist-test-slave-z8x9"
server_key: "e7921a8649ec724057c9bbc103ab295a"
server_key_iv: "7f0b64a45efdadcbfd867160ce0cc309"
server_key_version: "encryptionkey@0"
I20260807 23:02:35.896311 31351 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786143361779964-25976-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786143361779964-25976-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786143361779964-25976-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 23:02:35.909628 31351 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 23:02:35.914113 31351 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 23:02:35.914419 31351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 23:02:35.933883 31351 rpc_server.cc:307] RPC server started. Bound to: 127.25.94.60:36589
I20260807 23:02:35.933902 31453 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.94.60:36589 every 8 connection(s)
I20260807 23:02:35.935254 31351 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786143361779964-25976-0/minicluster-data/master-2/data/info.pb
I20260807 23:02:35.938910 31454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 23:02:35.940474 25976 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknHbun_/build/asan/bin/kudu as pid 31351
I20260807 23:02:35.940678 25976 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknHbun_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786143361779964-25976-0/minicluster-data/master-2/wal/instance
I20260807 23:02:35.941479 25976 external_mini_cluster.cc:1428] Setting key cdb830ac63c6586a7de391eb29810370
I20260807 23:02:35.946497 31454 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.94.62" port: 40737 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 23:02:35 dist-test-slave-z8x9 krb5kdc[31120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143754, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.60@KRBTEST.COM
Aug 07 23:02:35 dist-test-slave-z8x9 krb5kdc[31120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143755, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.60@KRBTEST.COM for kudu/127.25.94.62@KRBTEST.COM
I20260807 23:02:35.966457 31462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:35.949501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51848 (local address 127.25.94.62:40737)
0807 23:02:35.950775 (+ 1274us) server_negotiation.cc:206] Beginning negotiation
0807 23:02:35.950783 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 23:02:35.950803 (+ 20us) server_negotiation.cc:407] Connection header received
0807 23:02:35.950880 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:02:35.950884 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:02:35.950946 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:02:35.951027 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:02:35.952502 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:35.953626 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:35.955671 (+ 2045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:35.956073 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:35.956174 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:02:35.959654 (+ 3480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:02:35.959663 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:02:35.959670 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 23:02:35.959711 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:02:35.962921 (+ 3210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:02:35.963972 (+ 1051us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:02:35.963977 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:02:35.963979 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:02:35.964156 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:02:35.964563 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:02:35.964568 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:02:35.964571 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:02:35.964924 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:02:35.965093 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0807 23:02:35.965824 (+ 731us) server_negotiation.cc:299] Negotiation successful
0807 23:02:35.966061 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1143,"thread_start_us":1015,"threads_started":1}
I20260807 23:02:35.967486 31460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:35.943317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53108 (local address 127.25.94.60:36589)
0807 23:02:35.944233 (+ 916us) server_negotiation.cc:206] Beginning negotiation
0807 23:02:35.944244 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 23:02:35.944285 (+ 41us) server_negotiation.cc:407] Connection header received
0807 23:02:35.946060 (+ 1775us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:02:35.946082 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:02:35.946455 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:02:35.946789 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:02:35.948650 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:35.950658 (+ 2008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:35.952485 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:35.953579 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:35.953797 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:02:35.958474 (+ 4677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:02:35.958481 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:02:35.958493 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 23:02:35.958531 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:02:35.962304 (+ 3773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:02:35.963311 (+ 1007us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:02:35.963323 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:02:35.963325 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:02:35.963496 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:02:35.964132 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:02:35.964138 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:02:35.964141 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:02:35.964946 (+ 805us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:02:35.965315 (+ 369us) server_negotiation.cc:1035] Waiting for connection context
0807 23:02:35.965700 (+ 385us) server_negotiation.cc:299] Negotiation successful
0807 23:02:35.966161 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":44,"server-negotiator.queue_time_us":684,"thread_start_us":414,"threads_started":1}
I20260807 23:02:35.967504 31461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:35.949452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.62:40737 (local address 127.0.0.1:51848)
0807 23:02:35.950322 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0807 23:02:35.950376 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0807 23:02:35.950546 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 23:02:35.951127 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 23:02:35.951134 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 23:02:35.951937 (+ 803us) client_negotiation.cc:189] Negotiated authn=SASL
0807 23:02:35.952265 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:02:35.952277 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:02:35.953793 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:02:35.953798 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:02:35.955492 (+ 1694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:02:35.955507 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:02:35.956307 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:02:35.956314 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:02:35.956548 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:02:35.957307 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 23:02:35.957330 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0807 23:02:35.959441 (+ 2111us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 23:02:35.963051 (+ 3610us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 23:02:35.963065 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 23:02:35.963084 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0807 23:02:35.963823 (+ 739us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 23:02:35.964288 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 23:02:35.964291 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 23:02:35.964294 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 23:02:35.964449 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 23:02:35.965115 (+ 666us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 23:02:35.965123 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 23:02:35.965388 (+ 265us) client_negotiation.cc:769] Sending connection context
0807 23:02:35.965698 (+ 310us) client_negotiation.cc:240] Negotiation successful
0807 23:02:35.966376 (+ 678us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"mutex_wait_us":208,"thread_start_us":335,"threads_started":1}
I20260807 23:02:35.969666 31454 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.94.61" port: 37133 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 23:02:35 dist-test-slave-z8x9 krb5kdc[31120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143755, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.60@KRBTEST.COM for kudu/127.25.94.61@KRBTEST.COM
Aug 07 23:02:35 dist-test-slave-z8x9 krb5kdc[31120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143754, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.62@KRBTEST.COM
I20260807 23:02:35.985069 31461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:35.970514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.61:37133 (local address 127.0.0.1:49600)
0807 23:02:35.970696 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0807 23:02:35.970718 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 23:02:35.970855 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 23:02:35.971627 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 23:02:35.971633 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 23:02:35.972113 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0807 23:02:35.972255 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:02:35.972264 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:02:35.973696 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:02:35.973703 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:02:35.975015 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:02:35.975028 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:02:35.975962 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:02:35.975966 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:02:35.976114 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:02:35.976744 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 23:02:35.976770 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0807 23:02:35.979276 (+ 2506us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 23:02:35.982595 (+ 3319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 23:02:35.982601 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 23:02:35.982605 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 23:02:35.982994 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 23:02:35.983397 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 23:02:35.983402 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 23:02:35.983406 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 23:02:35.983572 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 23:02:35.984093 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 23:02:35.984098 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 23:02:35.984249 (+ 151us) client_negotiation.cc:769] Sending connection context
0807 23:02:35.984406 (+ 157us) client_negotiation.cc:240] Negotiation successful
0807 23:02:35.984695 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260807 23:02:35.985096 31346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:35.970501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49600 (local address 127.25.94.61:37133)
0807 23:02:35.970714 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0807 23:02:35.970720 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 23:02:35.970822 (+ 102us) server_negotiation.cc:407] Connection header received
0807 23:02:35.970986 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:02:35.970993 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:02:35.971082 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:02:35.971191 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:02:35.972433 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:35.973532 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:35.975167 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:35.975812 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:35.975954 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:02:35.979493 (+ 3539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:02:35.979501 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:02:35.979506 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 23:02:35.979559 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:02:35.982442 (+ 2883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:02:35.983121 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:02:35.983128 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:02:35.983132 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:02:35.983277 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:02:35.983687 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:02:35.983693 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:02:35.983696 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:02:35.983968 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:02:35.984126 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0807 23:02:35.984497 (+ 371us) server_negotiation.cc:299] Negotiation successful
0807 23:02:35.984711 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260807 23:02:35.986342 31454 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.94.60" port: 36589 } has no permanent_uuid. Determining permanent_uuid...
I20260807 23:02:35.987614 31462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:35.971187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51854 (local address 127.25.94.62:40737)
0807 23:02:35.971370 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0807 23:02:35.971375 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 23:02:35.971393 (+ 18us) server_negotiation.cc:407] Connection header received
0807 23:02:35.971582 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:02:35.971588 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:02:35.971672 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:02:35.971776 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:02:35.973430 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:35.974583 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:35.976338 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:35.976921 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:35.977045 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:02:35.982008 (+ 4963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:02:35.982012 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:02:35.982015 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 23:02:35.982047 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:02:35.984684 (+ 2637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:02:35.985494 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:02:35.985499 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:02:35.985502 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:02:35.985648 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:02:35.986126 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:02:35.986130 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:02:35.986132 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:02:35.986403 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:02:35.986695 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0807 23:02:35.987034 (+ 339us) server_negotiation.cc:299] Negotiation successful
0807 23:02:35.987284 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260807 23:02:35.992092 31235 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.94.60" port: 36589 } attempt: 4
Aug 07 23:02:35 dist-test-slave-z8x9 krb5kdc[31120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143755, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.60@KRBTEST.COM for kudu/127.25.94.60@KRBTEST.COM
Aug 07 23:02:35 dist-test-slave-z8x9 krb5kdc[31120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143754, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.94.61@KRBTEST.COM
Aug 07 23:02:35 dist-test-slave-z8x9 krb5kdc[31120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143754, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.62@KRBTEST.COM for kudu/127.25.94.60@KRBTEST.COM
I20260807 23:02:36.003708 31461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:35.987283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.60:36589 (local address 127.0.0.1:53112)
0807 23:02:35.987531 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0807 23:02:35.987547 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 23:02:35.987621 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 23:02:35.988118 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 23:02:35.988122 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 23:02:35.988562 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0807 23:02:35.988735 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:02:35.988744 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:02:35.990144 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:02:35.990152 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:02:35.991260 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:02:35.991277 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:02:35.992009 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:02:35.992050 (+ 41us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:02:35.992273 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:02:35.993405 (+ 1132us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 23:02:35.993431 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0807 23:02:35.996280 (+ 2849us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 23:02:36.000591 (+ 4311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 23:02:36.000595 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 23:02:36.000599 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 23:02:36.001165 (+ 566us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 23:02:36.001677 (+ 512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 23:02:36.001683 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 23:02:36.001686 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 23:02:36.001873 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 23:02:36.002509 (+ 636us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 23:02:36.002518 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 23:02:36.002680 (+ 162us) client_negotiation.cc:769] Sending connection context
0807 23:02:36.002842 (+ 162us) client_negotiation.cc:240] Negotiation successful
0807 23:02:36.003209 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149}
I20260807 23:02:36.004125 31460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:35.987215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53112 (local address 127.25.94.60:36589)
0807 23:02:35.987579 (+ 364us) server_negotiation.cc:206] Beginning negotiation
0807 23:02:35.987588 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 23:02:35.987609 (+ 21us) server_negotiation.cc:407] Connection header received
0807 23:02:35.987931 (+ 322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:02:35.987934 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:02:35.987989 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:02:35.988105 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:02:35.988870 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:35.990025 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:35.991418 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:35.991861 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:35.991996 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:02:35.996494 (+ 4498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:02:35.996503 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:02:35.996509 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 23:02:35.996551 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:02:36.000430 (+ 3879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:02:36.001360 (+ 930us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:02:36.001367 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:02:36.001373 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:02:36.001544 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:02:36.001981 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:02:36.001986 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:02:36.001989 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:02:36.002364 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:02:36.002522 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0807 23:02:36.002951 (+ 429us) server_negotiation.cc:299] Negotiation successful
0807 23:02:36.003420 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":185,"server-negotiator.queue_time_us":63}
I20260807 23:02:36.007035 31346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:35.989227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49604 (local address 127.25.94.61:37133)
0807 23:02:35.989377 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0807 23:02:35.989382 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 23:02:35.989399 (+ 17us) server_negotiation.cc:407] Connection header received
0807 23:02:35.989452 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:02:35.989457 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:02:35.989523 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:02:35.989619 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:02:35.990753 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:35.991690 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:35.993680 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:35.994349 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:35.994474 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:02:36.000151 (+ 5677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:02:36.000160 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:02:36.000164 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 23:02:36.000202 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:02:36.003704 (+ 3502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:02:36.004573 (+ 869us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:02:36.004579 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:02:36.004583 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:02:36.004731 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:02:36.005219 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:02:36.005225 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:02:36.005229 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:02:36.005558 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:02:36.005845 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0807 23:02:36.006421 (+ 576us) server_negotiation.cc:299] Negotiation successful
0807 23:02:36.006673 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260807 23:02:36.009660 31238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:35.992714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.60:36589 (local address 127.0.0.1:53128)
0807 23:02:35.992913 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0807 23:02:35.992934 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 23:02:35.993042 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 23:02:35.993940 (+ 898us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 23:02:35.993945 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 23:02:35.994426 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0807 23:02:35.994631 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:02:35.994643 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:02:35.996528 (+ 1885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:02:35.996533 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:02:35.998015 (+ 1482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:02:35.998028 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:02:35.998886 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:02:35.998893 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:02:35.999114 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:02:35.999694 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 23:02:35.999719 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0807 23:02:36.002707 (+ 2988us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 23:02:36.006703 (+ 3996us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 23:02:36.006708 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 23:02:36.006712 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 23:02:36.007221 (+ 509us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 23:02:36.007779 (+ 558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 23:02:36.007784 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 23:02:36.007788 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 23:02:36.007965 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 23:02:36.008610 (+ 645us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 23:02:36.008615 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 23:02:36.008771 (+ 156us) client_negotiation.cc:769] Sending connection context
0807 23:02:36.008926 (+ 155us) client_negotiation.cc:240] Negotiation successful
0807 23:02:36.009249 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260807 23:02:36.009707 31464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:35.992787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53128 (local address 127.25.94.60:36589)
0807 23:02:35.993581 (+ 794us) server_negotiation.cc:206] Beginning negotiation
0807 23:02:35.993589 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 23:02:35.993612 (+ 23us) server_negotiation.cc:407] Connection header received
0807 23:02:35.993660 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:02:35.993668 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:02:35.993742 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:02:35.994777 (+ 1035us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:02:35.994997 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:35.996348 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:35.998167 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:35.998723 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:35.998865 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:02:36.002981 (+ 4116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:02:36.002988 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:02:36.002992 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 23:02:36.003025 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:02:36.006546 (+ 3521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:02:36.007448 (+ 902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:02:36.007453 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:02:36.007456 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:02:36.007656 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:02:36.008089 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:02:36.008095 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:02:36.008098 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:02:36.008471 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:02:36.008655 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0807 23:02:36.009021 (+ 366us) server_negotiation.cc:299] Negotiation successful
0807 23:02:36.009252 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":670,"thread_start_us":433,"threads_started":1}
I20260807 23:02:36.010933 31454 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85e4774ac2f743048053be25c8281bd8: Bootstrap starting.
I20260807 23:02:36.012655 25976 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 23:02:36.014601 31235 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8ef131160eb445ba860c4129128a5fb: Bootstrap starting.
I20260807 23:02:36.014822 31454 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 85e4774ac2f743048053be25c8281bd8: Neither blocks nor log segments found. Creating new log.
I20260807 23:02:36.016106 31454 log.cc:824] T 00000000000000000000000000000000 P 85e4774ac2f743048053be25c8281bd8: Log is configured to *not* fsync() on all Append() calls
I20260807 23:02:36.018851 31235 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c8ef131160eb445ba860c4129128a5fb: Neither blocks nor log segments found. Creating new log.
I20260807 23:02:36.019045 31454 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85e4774ac2f743048053be25c8281bd8: No bootstrap required, opened a new log
I20260807 23:02:36.020268 31235 log.cc:824] T 00000000000000000000000000000000 P c8ef131160eb445ba860c4129128a5fb: Log is configured to *not* fsync() on all Append() calls
I20260807 23:02:36.015926 31200 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "c8ef131160eb445ba860c4129128a5fb"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51854
I20260807 23:02:36.023644 31235 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8ef131160eb445ba860c4129128a5fb: No bootstrap required, opened a new log
I20260807 23:02:36.026369 31454 raft_consensus.cc:348] T 00000000000000000000000000000000 P 85e4774ac2f743048053be25c8281bd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8ef131160eb445ba860c4129128a5fb" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 40737 } } peers { permanent_uuid: "b560dd8a492f49609945578af494f2ab" member_type: VOTER last_known_addr { host: "127.25.94.61" port: 37133 } } peers { permanent_uuid: "85e4774ac2f743048053be25c8281bd8" member_type: VOTER last_known_addr { host: "127.25.94.60" port: 36589 } }
I20260807 23:02:36.026741 31454 raft_consensus.cc:729] T 00000000000000000000000000000000 P 85e4774ac2f743048053be25c8281bd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85e4774ac2f743048053be25c8281bd8, State: Initialized, Role: FOLLOWER
I20260807 23:02:36.027192 31454 consensus_queue.cc:260] T 00000000000000000000000000000000 P 85e4774ac2f743048053be25c8281bd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8ef131160eb445ba860c4129128a5fb" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 40737 } } peers { permanent_uuid: "b560dd8a492f49609945578af494f2ab" member_type: VOTER last_known_addr { host: "127.25.94.61" port: 37133 } } peers { permanent_uuid: "85e4774ac2f743048053be25c8281bd8" member_type: VOTER last_known_addr { host: "127.25.94.60" port: 36589 } }
I20260807 23:02:36.028160 31468 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85e4774ac2f743048053be25c8281bd8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8ef131160eb445ba860c4129128a5fb" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 40737 } } peers { permanent_uuid: "b560dd8a492f49609945578af494f2ab" member_type: VOTER last_known_addr { host: "127.25.94.61" port: 37133 } } peers { permanent_uuid: "85e4774ac2f743048053be25c8281bd8" member_type: VOTER last_known_addr { host: "127.25.94.60" port: 36589 } } }
I20260807 23:02:36.028585 31468 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85e4774ac2f743048053be25c8281bd8 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 23:02:36.029423 31454 sys_catalog.cc:564] T 00000000000000000000000000000000 P 85e4774ac2f743048053be25c8281bd8 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 23:02:36.031426 31235 raft_consensus.cc:348] T 00000000000000000000000000000000 P c8ef131160eb445ba860c4129128a5fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8ef131160eb445ba860c4129128a5fb" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 40737 } } peers { permanent_uuid: "b560dd8a492f49609945578af494f2ab" member_type: VOTER last_known_addr { host: "127.25.94.61" port: 37133 } } peers { permanent_uuid: "85e4774ac2f743048053be25c8281bd8" member_type: VOTER last_known_addr { host: "127.25.94.60" port: 36589 } }
I20260807 23:02:36.031838 31235 raft_consensus.cc:729] T 00000000000000000000000000000000 P c8ef131160eb445ba860c4129128a5fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8ef131160eb445ba860c4129128a5fb, State: Initialized, Role: FOLLOWER
I20260807 23:02:36.032374 31235 consensus_queue.cc:260] T 00000000000000000000000000000000 P c8ef131160eb445ba860c4129128a5fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8ef131160eb445ba860c4129128a5fb" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 40737 } } peers { permanent_uuid: "b560dd8a492f49609945578af494f2ab" member_type: VOTER last_known_addr { host: "127.25.94.61" port: 37133 } } peers { permanent_uuid: "85e4774ac2f743048053be25c8281bd8" member_type: VOTER last_known_addr { host: "127.25.94.60" port: 36589 } }
I20260807 23:02:36.033502 31474 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8ef131160eb445ba860c4129128a5fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8ef131160eb445ba860c4129128a5fb" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 40737 } } peers { permanent_uuid: "b560dd8a492f49609945578af494f2ab" member_type: VOTER last_known_addr { host: "127.25.94.61" port: 37133 } } peers { permanent_uuid: "85e4774ac2f743048053be25c8281bd8" member_type: VOTER last_known_addr { host: "127.25.94.60" port: 36589 } } }
I20260807 23:02:36.034052 31474 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8ef131160eb445ba860c4129128a5fb [sys.catalog]: This master's current role is: FOLLOWER
I20260807 23:02:36.034701 31235 sys_catalog.cc:564] T 00000000000000000000000000000000 P c8ef131160eb445ba860c4129128a5fb [sys.catalog]: configured and running, proceeding with master startup.
I20260807 23:02:36.034799 31200 raft_consensus.cc:482] T 00000000000000000000000000000000 P c8ef131160eb445ba860c4129128a5fb [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 23:02:36.035001 31200 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c8ef131160eb445ba860c4129128a5fb [term 0 FOLLOWER]: Advancing to term 1
I20260807 23:02:36.039198 31200 raft_consensus.cc:504] T 00000000000000000000000000000000 P c8ef131160eb445ba860c4129128a5fb [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8ef131160eb445ba860c4129128a5fb" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 40737 } } peers { permanent_uuid: "b560dd8a492f49609945578af494f2ab" member_type: VOTER last_known_addr { host: "127.25.94.61" port: 37133 } } peers { permanent_uuid: "85e4774ac2f743048053be25c8281bd8" member_type: VOTER last_known_addr { host: "127.25.94.60" port: 36589 } }
I20260807 23:02:36.042119 31420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c8ef131160eb445ba860c4129128a5fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "85e4774ac2f743048053be25c8281bd8"
I20260807 23:02:36.042708 31420 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 85e4774ac2f743048053be25c8281bd8 [term 0 FOLLOWER]: Advancing to term 1
I20260807 23:02:36.048008 31420 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 85e4774ac2f743048053be25c8281bd8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8ef131160eb445ba860c4129128a5fb in term 1.
I20260807 23:02:36.048986 31167 leader_election.cc:302] T 00000000000000000000000000000000 P c8ef131160eb445ba860c4129128a5fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85e4774ac2f743048053be25c8281bd8, c8ef131160eb445ba860c4129128a5fb; no voters:
I20260807 23:02:36.049731 31474 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c8ef131160eb445ba860c4129128a5fb [term 1 FOLLOWER]: Leader election won for term 1
I20260807 23:02:36.053032 31200 leader_election.cc:288] T 00000000000000000000000000000000 P c8ef131160eb445ba860c4129128a5fb [CANDIDATE]: Term 1 election: Requested vote from peers b560dd8a492f49609945578af494f2ab (127.25.94.61:37133), 85e4774ac2f743048053be25c8281bd8 (127.25.94.60:36589)
I20260807 23:02:36.043705 31351 master_runner.cc:186] Error getting master registration for 127.25.94.62:40737: OK, instance_id { permanent_uuid: "c8ef131160eb445ba860c4129128a5fb" instance_seqno: 1786143754845297 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260807 23:02:36.065776 31351 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260807 23:02:36.069983 31474 raft_consensus.cc:686] T 00000000000000000000000000000000 P c8ef131160eb445ba860c4129128a5fb [term 1 LEADER]: Becoming Leader. State: Replica: c8ef131160eb445ba860c4129128a5fb, State: Running, Role: LEADER
I20260807 23:02:36.070677 31474 consensus_queue.cc:237] T 00000000000000000000000000000000 P c8ef131160eb445ba860c4129128a5fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8ef131160eb445ba860c4129128a5fb" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 40737 } } peers { permanent_uuid: "b560dd8a492f49609945578af494f2ab" member_type: VOTER last_known_addr { host: "127.25.94.61" port: 37133 } } peers { permanent_uuid: "85e4774ac2f743048053be25c8281bd8" member_type: VOTER last_known_addr { host: "127.25.94.60" port: 36589 } }
W20260807 23:02:36.073269 31494 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 85e4774ac2f743048053be25c8281bd8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 23:02:36.073452 31494 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 23:02:36.075006 31500 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8ef131160eb445ba860c4129128a5fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8ef131160eb445ba860c4129128a5fb. Latest consensus state: current_term: 1 leader_uuid: "c8ef131160eb445ba860c4129128a5fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8ef131160eb445ba860c4129128a5fb" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 40737 } } peers { permanent_uuid: "b560dd8a492f49609945578af494f2ab" member_type: VOTER last_known_addr { host: "127.25.94.61" port: 37133 } } peers { permanent_uuid: "85e4774ac2f743048053be25c8281bd8" member_type: VOTER last_known_addr { host: "127.25.94.60" port: 36589 } } }
I20260807 23:02:36.075516 31500 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8ef131160eb445ba860c4129128a5fb [sys.catalog]: This master's current role is: LEADER
I20260807 23:02:36.077085 31521 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
W20260807 23:02:36.085357 31169 tablet.cc:2369] T 00000000000000000000000000000000 P c8ef131160eb445ba860c4129128a5fb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 23:02:36.089149 31521 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 23:02:36.128729 31341 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.94.60" port: 36589 } attempt: 5
I20260807 23:02:36.085546 31139 master_runner.cc:186] Error getting master registration for 127.25.94.61:37133: OK, instance_id { permanent_uuid: "b560dd8a492f49609945578af494f2ab" instance_seqno: 1786143755390487 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260807 23:02:36.142755 31139 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Aug 07 23:02:36 dist-test-slave-z8x9 krb5kdc[31120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143755, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.94.61@KRBTEST.COM for kudu/127.25.94.60@KRBTEST.COM
I20260807 23:02:36.163756 31460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:36.130658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53150 (local address 127.25.94.60:36589)
0807 23:02:36.130921 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0807 23:02:36.130931 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 23:02:36.130959 (+ 28us) server_negotiation.cc:407] Connection header received
0807 23:02:36.131040 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:02:36.131047 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:02:36.131158 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:02:36.131274 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:02:36.132785 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:36.134305 (+ 1520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:36.136755 (+ 2450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:36.137848 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:36.138402 (+ 554us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:02:36.150625 (+ 12223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:02:36.150635 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:02:36.150642 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 23:02:36.150711 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:02:36.156803 (+ 6092us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:02:36.158238 (+ 1435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:02:36.158246 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:02:36.158250 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:02:36.158596 (+ 346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:02:36.159452 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:02:36.159458 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:02:36.159461 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:02:36.160258 (+ 797us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:02:36.160532 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0807 23:02:36.161595 (+ 1063us) server_negotiation.cc:299] Negotiation successful
0807 23:02:36.162182 (+ 587us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260807 23:02:36.164662 31345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:36.129682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.94.60:36589 (local address 127.0.0.1:53150)
0807 23:02:36.130214 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0807 23:02:36.130240 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0807 23:02:36.130387 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 23:02:36.131528 (+ 1141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 23:02:36.131536 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 23:02:36.132266 (+ 730us) client_negotiation.cc:189] Negotiated authn=SASL
0807 23:02:36.132542 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:02:36.132559 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:02:36.134570 (+ 2011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:02:36.134580 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:02:36.136500 (+ 1920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 23:02:36.136523 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 23:02:36.138840 (+ 2317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 23:02:36.138854 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 23:02:36.139725 (+ 871us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:02:36.140698 (+ 973us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 23:02:36.140754 (+ 56us) client_negotiation.cc:656] calling sasl_client_start()
0807 23:02:36.150308 (+ 9554us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 23:02:36.157068 (+ 6760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 23:02:36.157077 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 23:02:36.157084 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0807 23:02:36.158047 (+ 963us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 23:02:36.158873 (+ 826us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 23:02:36.158880 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 23:02:36.158884 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 23:02:36.159205 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 23:02:36.160728 (+ 1523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 23:02:36.160734 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 23:02:36.161029 (+ 295us) client_negotiation.cc:769] Sending connection context
0807 23:02:36.162112 (+ 1083us) client_negotiation.cc:240] Negotiation successful
0807 23:02:36.163437 (+ 1325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":353}
I20260807 23:02:36.167408 31346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:36.094992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49606 (local address 127.25.94.61:37133)
0807 23:02:36.095205 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0807 23:02:36.095212 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 23:02:36.110627 (+ 15415us) server_negotiation.cc:407] Connection header received
0807 23:02:36.111006 (+ 379us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:02:36.111016 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:02:36.111141 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:02:36.111240 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:02:36.114642 (+ 3402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:36.116096 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:36.125355 (+ 9259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:36.126357 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:36.126514 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:02:36.141564 (+ 15050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:02:36.141573 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:02:36.141579 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 23:02:36.141632 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:02:36.147001 (+ 5369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:02:36.161860 (+ 14859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:02:36.161868 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:02:36.161871 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:02:36.162060 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:02:36.164292 (+ 2232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:02:36.164299 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:02:36.164302 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:02:36.165039 (+ 737us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:02:36.165320 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0807 23:02:36.166529 (+ 1209us) server_negotiation.cc:299] Negotiation successful
0807 23:02:36.166962 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260807 23:02:36.171388 31341 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b560dd8a492f49609945578af494f2ab: Bootstrap starting.
W20260807 23:02:36.173484 31386 tablet.cc:2369] T 00000000000000000000000000000000 P 85e4774ac2f743048053be25c8281bd8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 23:02:36.180311 31341 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b560dd8a492f49609945578af494f2ab: Neither blocks nor log segments found. Creating new log.
I20260807 23:02:36.183857 31341 log.cc:824] T 00000000000000000000000000000000 P b560dd8a492f49609945578af494f2ab: Log is configured to *not* fsync() on all Append() calls
I20260807 23:02:36.187865 31420 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 85e4774ac2f743048053be25c8281bd8 [term 1 FOLLOWER]: Refusing update from remote peer c8ef131160eb445ba860c4129128a5fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 23:02:36.189302 31474 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c8ef131160eb445ba860c4129128a5fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "85e4774ac2f743048053be25c8281bd8" member_type: VOTER last_known_addr { host: "127.25.94.60" port: 36589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 23:02:36.192144 31462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:36.088731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51870 (local address 127.25.94.62:40737)
0807 23:02:36.088978 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0807 23:02:36.088986 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 23:02:36.119146 (+ 30160us) server_negotiation.cc:407] Connection header received
0807 23:02:36.120615 (+ 1469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:02:36.120624 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:02:36.120745 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:02:36.120849 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:02:36.143001 (+ 22152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:36.144430 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:36.150466 (+ 6036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:36.151357 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:36.151487 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:02:36.170909 (+ 19422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:02:36.170919 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:02:36.170924 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 23:02:36.170984 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:02:36.175434 (+ 4450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:02:36.183781 (+ 8347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:02:36.183792 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:02:36.183798 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:02:36.184308 (+ 510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:02:36.187741 (+ 3433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:02:36.187748 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:02:36.187751 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:02:36.188380 (+ 629us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:02:36.188867 (+ 487us) server_negotiation.cc:1035] Waiting for connection context
0807 23:02:36.191017 (+ 2150us) server_negotiation.cc:299] Negotiation successful
0807 23:02:36.191302 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260807 23:02:36.194152 31464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:36.102612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53144 (local address 127.25.94.60:36589)
0807 23:02:36.103174 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0807 23:02:36.103327 (+ 153us) server_negotiation.cc:399] Waiting for connection header
0807 23:02:36.119580 (+ 16253us) server_negotiation.cc:407] Connection header received
0807 23:02:36.120076 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:02:36.120085 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:02:36.120228 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:02:36.120342 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:02:36.146106 (+ 25764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:36.147398 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:36.153388 (+ 5990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:36.154193 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:36.154332 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:02:36.176833 (+ 22501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 23:02:36.176844 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 23:02:36.176850 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 23:02:36.176908 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0807 23:02:36.182560 (+ 5652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:02:36.186854 (+ 4294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:02:36.186864 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:02:36.186869 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:02:36.187103 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 23:02:36.189493 (+ 2390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 23:02:36.189499 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 23:02:36.189502 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 23:02:36.190101 (+ 599us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 23:02:36.190360 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0807 23:02:36.193440 (+ 3080us) server_negotiation.cc:299] Negotiation successful
0807 23:02:36.193720 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260807 23:02:36.202261 31341 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b560dd8a492f49609945578af494f2ab: No bootstrap required, opened a new log
I20260807 23:02:36.215270 31341 raft_consensus.cc:348] T 00000000000000000000000000000000 P b560dd8a492f49609945578af494f2ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8ef131160eb445ba860c4129128a5fb" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 40737 } } peers { permanent_uuid: "b560dd8a492f49609945578af494f2ab" member_type: VOTER last_known_addr { host: "127.25.94.61" port: 37133 } } peers { permanent_uuid: "85e4774ac2f743048053be25c8281bd8" member_type: VOTER last_known_addr { host: "127.25.94.60" port: 36589 } }
I20260807 23:02:36.215785 31341 raft_consensus.cc:729] T 00000000000000000000000000000000 P b560dd8a492f49609945578af494f2ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b560dd8a492f49609945578af494f2ab, State: Initialized, Role: FOLLOWER
I20260807 23:02:36.216385 31341 consensus_queue.cc:260] T 00000000000000000000000000000000 P b560dd8a492f49609945578af494f2ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8ef131160eb445ba860c4129128a5fb" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 40737 } } peers { permanent_uuid: "b560dd8a492f49609945578af494f2ab" member_type: VOTER last_known_addr { host: "127.25.94.61" port: 37133 } } peers { permanent_uuid: "85e4774ac2f743048053be25c8281bd8" member_type: VOTER last_known_addr { host: "127.25.94.60" port: 36589 } }
I20260807 23:02:36.218809 31468 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85e4774ac2f743048053be25c8281bd8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8ef131160eb445ba860c4129128a5fb. Latest consensus state: current_term: 1 leader_uuid: "c8ef131160eb445ba860c4129128a5fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8ef131160eb445ba860c4129128a5fb" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 40737 } } peers { permanent_uuid: "b560dd8a492f49609945578af494f2ab" member_type: VOTER last_known_addr { host: "127.25.94.61" port: 37133 } } peers { permanent_uuid: "85e4774ac2f743048053be25c8281bd8" member_type: VOTER last_known_addr { host: "127.25.94.60" port: 36589 } } }
I20260807 23:02:36.219161 31468 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85e4774ac2f743048053be25c8281bd8 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 23:02:36.221956 31529 mvcc.cc:205] Tried to move back new op lower bound from 7316044825232691200 to 7316044824871198720. Current Snapshot: MvccSnapshot[applied={T|T < 7316044825232691200}]
I20260807 23:02:36.224757 31531 mvcc.cc:205] Tried to move back new op lower bound from 7316044825232691200 to 7316044824871198720. Current Snapshot: MvccSnapshot[applied={T|T < 7316044825232691200}]
I20260807 23:02:36.226980 31468 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85e4774ac2f743048053be25c8281bd8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c8ef131160eb445ba860c4129128a5fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8ef131160eb445ba860c4129128a5fb" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 40737 } } peers { permanent_uuid: "b560dd8a492f49609945578af494f2ab" member_type: VOTER last_known_addr { host: "127.25.94.61" port: 37133 } } peers { permanent_uuid: "85e4774ac2f743048053be25c8281bd8" member_type: VOTER last_known_addr { host: "127.25.94.60" port: 36589 } } }
I20260807 23:02:36.227267 31468 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85e4774ac2f743048053be25c8281bd8 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 23:02:36.233727 31536 sys_catalog.cc:455] T 00000000000000000000000000000000 P b560dd8a492f49609945578af494f2ab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8ef131160eb445ba860c4129128a5fb" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 40737 } } peers { permanent_uuid: "b560dd8a492f49609945578af494f2ab" member_type: VOTER last_known_addr { host: "127.25.94.61" port: 37133 } } peers { permanent_uuid: "85e4774ac2f743048053be25c8281bd8" member_type: VOTER last_known_addr { host: "127.25.94.60" port: 36589 } } }
I20260807 23:02:36.234221 31536 sys_catalog.cc:458] T 00000000000000000000000000000000 P b560dd8a492f49609945578af494f2ab [sys.catalog]: This master's current role is: FOLLOWER
I20260807 23:02:36.236331 31341 sys_catalog.cc:564] T 00000000000000000000000000000000 P b560dd8a492f49609945578af494f2ab [sys.catalog]: configured and running, proceeding with master startup.
I20260807 23:02:36.236886 31302 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b560dd8a492f49609945578af494f2ab [term 0 FOLLOWER]: Advancing to term 1
I20260807 23:02:36.237454 31303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c8ef131160eb445ba860c4129128a5fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "b560dd8a492f49609945578af494f2ab"
I20260807 23:02:36.244146 31302 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b560dd8a492f49609945578af494f2ab [term 1 FOLLOWER]: Refusing update from remote peer c8ef131160eb445ba860c4129128a5fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 23:02:36.249085 31303 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b560dd8a492f49609945578af494f2ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8ef131160eb445ba860c4129128a5fb in term 1.
I20260807 23:02:36.254738 31474 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c8ef131160eb445ba860c4129128a5fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b560dd8a492f49609945578af494f2ab" member_type: VOTER last_known_addr { host: "127.25.94.61" port: 37133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 23:02:36.264912 31474 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8ef131160eb445ba860c4129128a5fb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c8ef131160eb445ba860c4129128a5fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8ef131160eb445ba860c4129128a5fb" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 40737 } } peers { permanent_uuid: "b560dd8a492f49609945578af494f2ab" member_type: VOTER last_known_addr { host: "127.25.94.61" port: 37133 } } peers { permanent_uuid: "85e4774ac2f743048053be25c8281bd8" member_type: VOTER last_known_addr { host: "127.25.94.60" port: 36589 } } }
I20260807 23:02:36.265309 31474 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8ef131160eb445ba860c4129128a5fb [sys.catalog]: This master's current role is: LEADER
I20260807 23:02:36.266213 31521 catalog_manager.cc:1347] Generated new cluster ID: 835dc04820df4d2da61d446a7e0cd0dc
I20260807 23:02:36.266335 31521 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 23:02:36.317463 31536 sys_catalog.cc:455] T 00000000000000000000000000000000 P b560dd8a492f49609945578af494f2ab [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8ef131160eb445ba860c4129128a5fb. Latest consensus state: current_term: 1 leader_uuid: "c8ef131160eb445ba860c4129128a5fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8ef131160eb445ba860c4129128a5fb" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 40737 } } peers { permanent_uuid: "b560dd8a492f49609945578af494f2ab" member_type: VOTER last_known_addr { host: "127.25.94.61" port: 37133 } } peers { permanent_uuid: "85e4774ac2f743048053be25c8281bd8" member_type: VOTER last_known_addr { host: "127.25.94.60" port: 36589 } } }
I20260807 23:02:36.317485 31552 sys_catalog.cc:455] T 00000000000000000000000000000000 P b560dd8a492f49609945578af494f2ab [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c8ef131160eb445ba860c4129128a5fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8ef131160eb445ba860c4129128a5fb" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 40737 } } peers { permanent_uuid: "b560dd8a492f49609945578af494f2ab" member_type: VOTER last_known_addr { host: "127.25.94.61" port: 37133 } } peers { permanent_uuid: "85e4774ac2f743048053be25c8281bd8" member_type: VOTER last_known_addr { host: "127.25.94.60" port: 36589 } } }
I20260807 23:02:36.317659 31500 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8ef131160eb445ba860c4129128a5fb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c8ef131160eb445ba860c4129128a5fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8ef131160eb445ba860c4129128a5fb" member_type: VOTER last_known_addr { host: "127.25.94.62" port: 40737 } } peers { permanent_uuid: "b560dd8a492f49609945578af494f2ab" member_type: VOTER last_known_addr { host: "127.25.94.61" port: 37133 } } peers { permanent_uuid: "85e4774ac2f743048053be25c8281bd8" member_type: VOTER last_known_addr { host: "127.25.94.60" port: 36589 } } }
I20260807 23:02:36.317920 31552 sys_catalog.cc:458] T 00000000000000000000000000000000 P b560dd8a492f49609945578af494f2ab [sys.catalog]: This master's current role is: FOLLOWER
I20260807 23:02:36.317925 31500 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8ef131160eb445ba860c4129128a5fb [sys.catalog]: This master's current role is: LEADER
I20260807 23:02:36.317917 31536 sys_catalog.cc:458] T 00000000000000000000000000000000 P b560dd8a492f49609945578af494f2ab [sys.catalog]: This master's current role is: FOLLOWER
I20260807 23:02:36.324074 31521 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 23:02:36.326727 31521 catalog_manager.cc:1504] Loading token signing keys...
I20260807 23:02:36.356995 31521 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c8ef131160eb445ba860c4129128a5fb: Generated new TSK 0
I20260807 23:02:36.358253 31521 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 23:02:36.381664 31560 catalog_manager.cc:1259] Loaded cluster ID: 835dc04820df4d2da61d446a7e0cd0dc
I20260807 23:02:36.381793 31560 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b560dd8a492f49609945578af494f2ab: loading cluster ID for follower catalog manager: success
I20260807 23:02:36.386101 31560 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b560dd8a492f49609945578af494f2ab: acquiring CA information for follower catalog manager: success
I20260807 23:02:36.389040 31560 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b560dd8a492f49609945578af494f2ab: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260807 23:02:36.524287 31462 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 23:02:36.505469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51872 (local address 127.25.94.62:40737)
0807 23:02:36.505702 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0807 23:02:36.505709 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 23:02:36.520079 (+ 14370us) server_negotiation.cc:407] Connection header received
0807 23:02:36.522582 (+ 2503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:02:36.522589 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:02:36.522680 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:02:36.522783 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:02:36.524021 (+ 1238us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51872: BlockingRecv error: recv got EOF from 127.0.0.1:51872 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260807 23:02:36.526165 31346 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 23:02:36.511805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49614 (local address 127.25.94.61:37133)
0807 23:02:36.512028 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0807 23:02:36.512035 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 23:02:36.522122 (+ 10087us) server_negotiation.cc:407] Connection header received
0807 23:02:36.522324 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:02:36.522330 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:02:36.522429 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:02:36.522545 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:02:36.525926 (+ 3381us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49614: BlockingRecv error: recv got EOF from 127.0.0.1:49614 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260807 23:02:36.530859 31464 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 23:02:36.518063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53154 (local address 127.25.94.60:36589)
0807 23:02:36.518311 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0807 23:02:36.518321 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 23:02:36.527126 (+ 8805us) server_negotiation.cc:407] Connection header received
0807 23:02:36.527313 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:02:36.527322 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:02:36.527450 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:02:36.527582 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:02:36.530553 (+ 2971us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53154: BlockingRecv error: recv got EOF from 127.0.0.1:53154 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260807 23:02:36.576822 31346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:36.560849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49620 (local address 127.25.94.61:37133)
0807 23:02:36.561057 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0807 23:02:36.561064 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 23:02:36.568642 (+ 7578us) server_negotiation.cc:407] Connection header received
0807 23:02:36.569236 (+ 594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:02:36.569244 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:02:36.569482 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:02:36.569658 (+ 176us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 23:02:36.570446 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:36.571629 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:36.573743 (+ 2114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:36.574405 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:36.574508 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:02:36.575289 (+ 781us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 23:02:36.575951 (+ 662us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 23:02:36.576065 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0807 23:02:36.576328 (+ 263us) server_negotiation.cc:299] Negotiation successful
0807 23:02:36.576427 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
W20260807 23:02:36.620911 31464 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 23:02:36.611283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53156 (local address 127.25.94.60:36589)
0807 23:02:36.611550 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0807 23:02:36.611557 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 23:02:36.618562 (+ 7005us) server_negotiation.cc:407] Connection header received
0807 23:02:36.619149 (+ 587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:02:36.619158 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:02:36.619310 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:02:36.619407 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:02:36.620632 (+ 1225us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53156: BlockingRecv error: recv got EOF from 127.0.0.1:53156 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
I20260807 23:02:36.678110 31346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:36.655725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49628 (local address 127.25.94.61:37133)
0807 23:02:36.655957 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0807 23:02:36.655964 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 23:02:36.669946 (+ 13982us) server_negotiation.cc:407] Connection header received
0807 23:02:36.670565 (+ 619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:02:36.670572 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:02:36.670671 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:02:36.670759 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 23:02:36.671492 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:36.672873 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:36.675388 (+ 2515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:36.676105 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:36.676240 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:02:36.677019 (+ 779us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 23:02:36.677267 (+ 248us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 23:02:36.677348 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0807 23:02:36.677601 (+ 253us) server_negotiation.cc:299] Negotiation successful
0807 23:02:36.677734 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
W20260807 23:02:36.727362 31464 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 23:02:36.711698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53164 (local address 127.25.94.60:36589)
0807 23:02:36.711920 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0807 23:02:36.711929 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 23:02:36.725476 (+ 13547us) server_negotiation.cc:407] Connection header received
0807 23:02:36.725799 (+ 323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:02:36.725805 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:02:36.725929 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:02:36.726014 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:02:36.727083 (+ 1069us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53164: BlockingRecv error: recv got EOF from 127.0.0.1:53164 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
I20260807 23:02:36.779525 31346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:36.766883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49640 (local address 127.25.94.61:37133)
0807 23:02:36.767088 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0807 23:02:36.767095 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 23:02:36.772745 (+ 5650us) server_negotiation.cc:407] Connection header received
0807 23:02:36.772959 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:02:36.772971 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:02:36.773074 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:02:36.773230 (+ 156us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 23:02:36.773741 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:36.774930 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:36.777313 (+ 2383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:36.777947 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:36.778081 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:02:36.778594 (+ 513us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 23:02:36.778757 (+ 163us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 23:02:36.778871 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0807 23:02:36.779041 (+ 170us) server_negotiation.cc:299] Negotiation successful
0807 23:02:36.779135 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
W20260807 23:02:36.791427 31464 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 23:02:36.787131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53174 (local address 127.25.94.60:36589)
0807 23:02:36.787337 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0807 23:02:36.787343 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 23:02:36.789738 (+ 2395us) server_negotiation.cc:407] Connection header received
0807 23:02:36.789922 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:02:36.789928 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:02:36.790021 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:02:36.790209 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:02:36.791202 (+ 993us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53174: BlockingRecv error: recv got EOF from 127.0.0.1:53174 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
I20260807 23:02:36.806653 31346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:36.799361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49650 (local address 127.25.94.61:37133)
0807 23:02:36.799632 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0807 23:02:36.799646 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0807 23:02:36.800278 (+ 632us) server_negotiation.cc:407] Connection header received
0807 23:02:36.800486 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:02:36.800492 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:02:36.800600 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:02:36.800713 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 23:02:36.801351 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:36.802484 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:36.804621 (+ 2137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:36.805096 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:36.805285 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:02:36.805750 (+ 465us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 23:02:36.805851 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 23:02:36.805926 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0807 23:02:36.806172 (+ 246us) server_negotiation.cc:299] Negotiation successful
0807 23:02:36.806272 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125}
W20260807 23:02:36.815738 31464 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 23:02:36.813312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53176 (local address 127.25.94.60:36589)
0807 23:02:36.813595 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0807 23:02:36.813602 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 23:02:36.814183 (+ 581us) server_negotiation.cc:407] Connection header received
0807 23:02:36.814454 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:02:36.814460 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:02:36.814547 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:02:36.814663 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:02:36.815454 (+ 791us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53176: BlockingRecv error: recv got EOF from 127.0.0.1:53176 (error 108)
Metrics: {"server-negotiator.queue_time_us":153}
I20260807 23:02:36.834529 31346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:36.827656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49652 (local address 127.25.94.61:37133)
0807 23:02:36.827868 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0807 23:02:36.827875 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 23:02:36.828264 (+ 389us) server_negotiation.cc:407] Connection header received
0807 23:02:36.828488 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:02:36.828492 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:02:36.828563 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:02:36.828651 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 23:02:36.829301 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:36.830116 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:36.832135 (+ 2019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:36.832694 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:36.832821 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:02:36.833576 (+ 755us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 23:02:36.833749 (+ 173us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 23:02:36.833861 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0807 23:02:36.834103 (+ 242us) server_negotiation.cc:299] Negotiation successful
0807 23:02:36.834208 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
W20260807 23:02:36.844458 31464 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 23:02:36.841769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53192 (local address 127.25.94.60:36589)
0807 23:02:36.841993 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0807 23:02:36.842000 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 23:02:36.842854 (+ 854us) server_negotiation.cc:407] Connection header received
0807 23:02:36.843010 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:02:36.843016 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:02:36.843121 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:02:36.843250 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:02:36.844204 (+ 954us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53192: BlockingRecv error: recv got EOF from 127.0.0.1:53192 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
I20260807 23:02:36.878011 31346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:36.871199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49664 (local address 127.25.94.61:37133)
0807 23:02:36.871405 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0807 23:02:36.871412 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 23:02:36.873019 (+ 1607us) server_negotiation.cc:407] Connection header received
0807 23:02:36.873293 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:02:36.873297 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:02:36.873371 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:02:36.873497 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 23:02:36.873948 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:36.874748 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:36.876100 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:36.876521 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:36.876661 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:02:36.877135 (+ 474us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 23:02:36.877346 (+ 211us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 23:02:36.877459 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0807 23:02:36.877621 (+ 162us) server_negotiation.cc:299] Negotiation successful
0807 23:02:36.877714 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260807 23:02:36.886565 31464 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 23:02:36.884493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53200 (local address 127.25.94.60:36589)
0807 23:02:36.884717 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0807 23:02:36.884723 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 23:02:36.885058 (+ 335us) server_negotiation.cc:407] Connection header received
0807 23:02:36.885352 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:02:36.885356 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:02:36.885426 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:02:36.885509 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:02:36.886342 (+ 833us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53200: BlockingRecv error: recv got EOF from 127.0.0.1:53200 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
I20260807 23:02:36.928880 31346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:36.922538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49680 (local address 127.25.94.61:37133)
0807 23:02:36.922749 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0807 23:02:36.922755 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 23:02:36.923090 (+ 335us) server_negotiation.cc:407] Connection header received
0807 23:02:36.923314 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:02:36.923320 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:02:36.923409 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:02:36.923521 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 23:02:36.923873 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:36.925083 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:36.926677 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:36.927471 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:36.927632 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:02:36.928057 (+ 425us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 23:02:36.928209 (+ 152us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 23:02:36.928298 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0807 23:02:36.928472 (+ 174us) server_negotiation.cc:299] Negotiation successful
0807 23:02:36.928555 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
W20260807 23:02:36.939330 31464 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 23:02:36.936239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53202 (local address 127.25.94.60:36589)
0807 23:02:36.936457 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0807 23:02:36.936467 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 23:02:36.937807 (+ 1340us) server_negotiation.cc:407] Connection header received
0807 23:02:36.937986 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:02:36.937993 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:02:36.938091 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:02:36.938234 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:02:36.939046 (+ 812us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53202: BlockingRecv error: recv got EOF from 127.0.0.1:53202 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
I20260807 23:02:37.011858 31346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:37.006610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49696 (local address 127.25.94.61:37133)
0807 23:02:37.006862 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0807 23:02:37.006869 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 23:02:37.007144 (+ 275us) server_negotiation.cc:407] Connection header received
0807 23:02:37.007313 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:02:37.007317 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:02:37.007403 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:02:37.007485 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 23:02:37.007909 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:37.008681 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:37.010117 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:37.010553 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:37.010656 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:02:37.011065 (+ 409us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 23:02:37.011163 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 23:02:37.011231 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0807 23:02:37.011447 (+ 216us) server_negotiation.cc:299] Negotiation successful
0807 23:02:37.011510 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260807 23:02:37.019444 31464 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 23:02:37.017453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53216 (local address 127.25.94.60:36589)
0807 23:02:37.017686 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0807 23:02:37.017694 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 23:02:37.018047 (+ 353us) server_negotiation.cc:407] Connection header received
0807 23:02:37.018266 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:02:37.018270 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:02:37.018360 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:02:37.018471 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0807 23:02:37.019241 (+ 770us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53216: BlockingRecv error: recv got EOF from 127.0.0.1:53216 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260807 23:02:37.075582 31494 catalog_manager.cc:1259] Loaded cluster ID: 835dc04820df4d2da61d446a7e0cd0dc
I20260807 23:02:37.075742 31494 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 85e4774ac2f743048053be25c8281bd8: loading cluster ID for follower catalog manager: success
I20260807 23:02:37.078383 31494 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 85e4774ac2f743048053be25c8281bd8: acquiring CA information for follower catalog manager: success
I20260807 23:02:37.080206 31494 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 85e4774ac2f743048053be25c8281bd8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260807 23:02:37.156791 31346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:37.151515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49698 (local address 127.25.94.61:37133)
0807 23:02:37.151780 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0807 23:02:37.151787 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 23:02:37.152014 (+ 227us) server_negotiation.cc:407] Connection header received
0807 23:02:37.152161 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:02:37.152165 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:02:37.152240 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:02:37.152347 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 23:02:37.152846 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:37.153590 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:37.155013 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:37.155491 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:37.155604 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:02:37.156036 (+ 432us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 23:02:37.156155 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 23:02:37.156225 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0807 23:02:37.156391 (+ 166us) server_negotiation.cc:299] Negotiation successful
0807 23:02:37.156460 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260807 23:02:37.168764 31464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:37.162908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53232 (local address 127.25.94.60:36589)
0807 23:02:37.163140 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0807 23:02:37.163148 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 23:02:37.163401 (+ 253us) server_negotiation.cc:407] Connection header received
0807 23:02:37.163627 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:02:37.163637 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:02:37.163788 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:02:37.163921 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 23:02:37.164338 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:37.165105 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:37.166455 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:37.166885 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:37.167027 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:02:37.167584 (+ 557us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 23:02:37.168111 (+ 527us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 23:02:37.168273 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0807 23:02:37.168390 (+ 117us) server_negotiation.cc:299] Negotiation successful
0807 23:02:37.168477 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260807 23:02:37.193275 31716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:37.175518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51880 (local address 127.25.94.62:40737)
0807 23:02:37.186032 (+ 10514us) server_negotiation.cc:206] Beginning negotiation
0807 23:02:37.186041 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 23:02:37.186073 (+ 32us) server_negotiation.cc:407] Connection header received
0807 23:02:37.186210 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:02:37.186217 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:02:37.186477 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:02:37.186682 (+ 205us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 23:02:37.187250 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:37.188400 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:37.190275 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:37.190702 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:37.190860 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:02:37.191631 (+ 771us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 23:02:37.192145 (+ 514us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 23:02:37.192260 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0807 23:02:37.192815 (+ 555us) server_negotiation.cc:299] Negotiation successful
0807 23:02:37.192933 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10361,"thread_start_us":10239,"threads_started":1}
I20260807 23:02:37.194072 31346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:37.175803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49708 (local address 127.25.94.61:37133)
0807 23:02:37.175993 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0807 23:02:37.175999 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 23:02:37.185909 (+ 9910us) server_negotiation.cc:407] Connection header received
0807 23:02:37.186495 (+ 586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:02:37.186503 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:02:37.186609 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:02:37.186723 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 23:02:37.188276 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:37.189084 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:37.191648 (+ 2564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:37.192354 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:37.192483 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:02:37.193020 (+ 537us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 23:02:37.193164 (+ 144us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 23:02:37.193297 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0807 23:02:37.193676 (+ 379us) server_negotiation.cc:299] Negotiation successful
0807 23:02:37.193773 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260807 23:02:37.194335 31464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 23:02:37.175580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53234 (local address 127.25.94.60:36589)
0807 23:02:37.176201 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0807 23:02:37.176210 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 23:02:37.186529 (+ 10319us) server_negotiation.cc:407] Connection header received
0807 23:02:37.187286 (+ 757us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 23:02:37.187290 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 23:02:37.187380 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 23:02:37.187450 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 23:02:37.188213 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:37.189352 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:37.191751 (+ 2399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 23:02:37.192228 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 23:02:37.192388 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 23:02:37.193054 (+ 666us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 23:02:37.193228 (+ 174us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 23:02:37.193342 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0807 23:02:37.193956 (+ 614us) server_negotiation.cc:299] Negotiation successful
0807 23:02:37.194045 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":446}
I20260807 23:02:37.199059 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 31139
I20260807 23:02:37.200342 31229 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 23:02:37.313515 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 31139
I20260807 23:02:37.340903 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 31242
I20260807 23:02:37.342168 31334 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 23:02:37.452885 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 31242
I20260807 23:02:37.480625 25976 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknHbun_/build/asan/bin/kudu pid 31351
I20260807 23:02:37.481875 31447 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 23:02:37.590130 25976 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknHbun_/build/asan/bin/kudu with pid 31351
2026-08-07T23:02:37Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4951 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (10211 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (395771 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST